NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan
DOI: https://doi.org/10.20736/0002001318

Overview of the NTCIR-17 FairWeb-1 Task
Sijie Tao, Nuo Chen,
Tetsuya Sakai
Waseda University, Japantsjmailbox@ruri.waseda.jp
pleviumtan@toki.waseda.jp
tetsuyasakai@acm.org

Zhumin Chu

Hiromi Arai

Tsinghua University, P.R.C.
chuzm19@mails.tsinghua.edu.cn

RIKEN AIP, Japanhiromi.arai@riken.jp

Ian Soboroff

Nicola Ferro

Maria Maistro

NIST, USAian.soboroff@nist.gov

University of Padua, Italy
ferro@dei.unipd.it

University of Copenhagen, Denmark
mm@di.ku.dk

ABSTRACTThe rest of this paper is organised as follows: Section 2 describes
the task specification, evaluation measures and the pilot data. Section 3 introduces the construction of the test collection, including
information regarding the target corpus, the submitted runs, andthe entity annotation. Section 4 shows the details of the official
evaluation results of the submitted runs. Finally, Section 5 conludes
the paper.

This paper provides an overview of the NTCIR-17 FairWeb-1 Task.
FairWeb-1 is an English web search task which seeks more thanan ad-hoc web search task. Our task considers not only document
relevance but also group fairness. We designed three types of search
topics for this task: researchers (R), movies (M), and Youtube contents (Y). For each topic type, attribute sets are defined for considering group fairness. We utilise a deduped version of the Chuweb21
corpus as the target corpus. We received 28 runs from six teams, including six runs from the organisers team. In this paper, we describe
the task, the test collection construction and the official evalution
results of the submitted runs.

1

Table 1: Timeline of FairWeb-1 at NTCIR-17

INTRODUCTION

This paper presents an overview of FairWeb-1, a pilot task at NTCIR17. Unlike Ad-hoc document retrieval tasks such as We Want Web(WWW) tasks [19] at previous NTCIR conferences where the participant teams only need to consider document relevance, FairWeb-1
considers not only document relevance from a viewpoint of searchengine users but also group fairness from a viewpoint of entities
that are being sought. Three entity types are considered at FairWeb1: researchers (R), movies (M), and Youtube contents (Y). Search
topics are designed to describe information needs about the three
types of entities. For each type of entities, we have one or two attribute sets, which contain either nominal or ordinal groups definedfor considering group fairness. Moreover, a target distribution isprovided for each attribute set. Given the search topics and theattribute sets, the participant teams are expecetd to submit SERPs1
that not only contain relevant documents at the top ranks but also
are group fair with respect to each attribute set. The submitted
runs are evaluated with a suite of evaluation measures called GFR(Group Fairness Relevance), which considers both relevance andgroup fairness [17].
Table 1 shows the timeline of the NTCIR-17 FairWeb-1 task. Table 2 shows the information of the participant teams and the numbers of submitted runs. Nine teams resgistered for the task, and by
the submission due, we received 28 runs from six participant teams,
including six runs from the organisers team. More information
about the task are available at http://sakailab.com/fairweb1/. Details of the runs can be found in the participants’ papers [3, 4, 6, 21].

Time

Content

October 3 2022
February 15 2023
March 15 2023
May 22 2023
May 23 - July 31 2023
August 1 2023
September 1 2023
November 1 2023
December 2023

Sample topics released
Pilot runs released
Topics released
Run submissions due
Entity annotations
Evaluation results released
Draft participant paper submissions due
Camera-ready paper submissions due
NTCIR-17 Conference@NII, Tokyo, Japan

Table 2: FairWeb-1 participants and the number of submitted runs.
Team name[undisclosed]∗
RSLFW [6]THUIR [21]UDinfo [3]rmit_ir[4]ORGANISERS

#runs
2
5
5
5
5
6
Total #runs
28∗The runs from this team contained bugs and are not useful forour analysis.

2Institution[undisclosed]
Waseda University, Japan
Tsinghua University, P.R.C.
University of Delaware, U.S.A.
RMIT University, Australia

TASK

FairWeb-1 is an English web search task which considers groupfairness. Imagine that we are serving as chairs of an IR conference,
and we want to hire diverse IR researchers as organisers. There

1 Search Engine Result Pages

284

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japanare several deminsions we may need consider: different career
stages (include junior researchers, not just famous researchers),
different genders, and different nationalities. However, if we searchfor “information retrieval researchers” with existing search engines,
they may return only famous people with a poor gender balance.
The motivation of this task is to encourage researchers develop
search algorithms that retrieve not only relevant but also group-fairresults.

2.1

Given a certain type of search topic and the attribute sets withtarget distributions, the participants are expected to return a SERP
that contains not only relevant documents near top ranks, butalso is group fair with respect to each attribute set. The submitted
runs are evaluated with a framework named GFR (Group Fairness
Relevance), which is a combination of relevance-based evaluation
measures and group fairness measures. At FairWeb-1, we use ERR
and iRBU for relevance evaluation. The group fairness of a run
is evaluated by measuring the similarity between the achieved
distribution and the target distribution. The details of evaluation
measures will be given in the next section.
Slides introducing the task can be found at https://waseda.box.
com/fairweb1intro2023feb.

Task Specification

FairWeb-1 considers not only relevance but also group fairness
from a viewpoint of entities that are being sought. Four entity types
were considered when the task was lauched: researhers (R), movies(M), Twitter accounts (T), and Youtube contents (Y). Based on the
results of pilot data experiments, we decided to drop Twitter (T)topics as it was difficult to locate a sufficient number of Twitter
account from retrieved documents. Therefore, we construct thetest collections based on three entity types. In our task, relevant
entities are entities that satisfy the condition specified in the topic
description. Topics are developed based on the four entity types, so
entity types can also be considered as topic types at FairWeb-1. For
example, for an M type topic “Daniel Craig 007 movies”, No Time
to Die is a relevant entity, while GoldenEye is an M type entity but
it is not relevant to this topic.
For each entity/topic type, one or two attribute sets are defined
to consider group fairness. Table 3 shows the attribute sets for each
topic type. For researchers topics (R topics), we have two attributesets: HINDEX (ordinal) and GENDER (nominal). HINDEX contains
groups of researchers’ based on Google scholar h-indexes. Take𝑥 as a researcher’s h-index, we consider four groups in HINDEX:
𝑥 < 10, 10 ≤ 𝑥 < 30, 30 ≤ 𝑥 < 50, and 50 ≤ 𝑥. For GENDER,
we consider three groups for convenience: he, she, and other. This
attribute is collected based soly on what pronoun is used in one of
the researcher’s official biographies that we have located. Note that
this label does not reflect our view on the gender of each researcher.
For movies topics (M topics), we have RATINGS (ordinal) andORIGIN (nominal). RATINGS contains groups based on the numberof ratings on the movie’s IMDb page.2 Take 𝑥 as the number ofratings, four groups are considered in this set: 𝑥 < 100, 100 ≤ 𝑥 <10, 000, 10, 000 ≤ 𝑥 < 1, 000, 000, and 1, 000, 000 ≤ 𝑥. ORIGIN contains eight geographic regions: Africa, America, Antractica, Asia,
Caribbean, Europe, Middle East, and Oceania. For each movie, the“country of origin” field on its IMDb page (which may contain multiple country/region names) is mapped to these eight regions. The
mapping from countries/regions to the ORIGIN groups can be foundin this excel file: https://waseda.box.com/ORIGIN-golddistribution.
For Youtube topics (Y topics), we have one attribute set named
SUBSCS (ordinal). Retrieved Youtube contents are divided into fourgroups based on the numbers of subscribers of the content creator.
The grouping strategy is the same as RATINGS of M topics.
For each attribute set, we used a uniform distribution as thetarget distribution, except for ORIGIN. For ORIGIN, we defineda target distribution based on how many countries/regions each
ORIGIN group contains; the distribution can also be found in the
ORIGIN-golddistribution excel file.

Table 3: Attribute sets for each topic type
Topic type
R
M
Y

2.2

Attribute sets
HINDEX (ordinal, 4 groups)
GENDER (nominal, 3 groups)
RATINGS (ordinal, 4 groups)
ORIGIN (nominal, 8 groups)
SUBSCS (ordinal, 4 groups)

Evaluation Measures

For evaluating the runs based on relevance and group fairness, we
utilise the GFR (Group Fairness and Relevance) framework [17];the details can be found in the paper. Below, we briefly explain how
the measures are computed for the FairWeb-1 task.
Let 𝐴 be an attribute set containing attribute values (or groups)
𝑎𝑖 (𝑖 = 1, . . . , |𝐴|), with a target distribution 𝑝 ∗ with group membership probabilities 𝑝 ∗ (𝑎𝑖 ). For example, for 𝐴 = GENDER ={he, she, other}, we have 𝑝 ∗ (he) = 𝑝 ∗ (she) = 𝑝 ∗ (other) = 1/3.
Given a ranked list 𝐿 (i.e., a SERP) of documents to be evaluated,
we require a group membership vector with respect to 𝐴 for eachdocument. More specifically, for the document ranked at 𝑘, its groupmembership vector contains as its elements 𝐺 (𝐿, 𝑘, 𝑎𝑖 ), the probaÍ
bility that the document belongs to group 𝑎𝑖 , s.t. 𝑖 𝐺 (𝐿, 𝑘, 𝑎𝑖 ) = 1.
(Section 3.5 explains how the vectors are derived from relevant
entity annotations in our task.) For example, a document may havea group membership vector of (1, 0, 0) for GENDER, which meansthat the document is 100% about researchers who uses the “he”
pronoun.
From the group membership vectors, we can compute the achieved
distribution 𝑝 𝐿,𝑘 at rank 𝑘 of 𝐿 with respect to 𝐴, where the probaÍ
bility for each group is given by 𝑝 𝐿,𝑘 (𝑎𝑖 ) = 𝑘𝑗=1 𝐺 (𝐿, 𝑗, 𝑎𝑖 )/𝑘, i.e.,
the average probability across ranks 1 through 𝑘. Then, for every
rank 𝑘 where there is a relevant document, we can compute a distribution similarity between the achived distribution at 𝑘 and thetarget distribution:
DistrSim(𝐿, 𝑘) = DistrSim(𝑝 𝐿,𝑘 ∥ 𝑝 ∗ ) = 1−Divergence(𝑝 𝐿,𝑘 ∥ 𝑝 ∗ ) ,(1)
where the Divergence function is JSD (Jensen-Shannon Divergence)
for attribute sets containing nominal groups (i.e., GENDER and ORIGIN in our case), and either NMD (Normalised Match Distance)

2 https://www.imdb.com/

285

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

or RNOD (Root Normalised Order-aware Divergence) for thosecontaining ordinal groups (i.e., HINDEX, RATINGS, and SUBSCS)
in our case. The details and properties of these divergence measures are discussed elsewhere [13, 14]. It should be noted that divergences such as JSD, Kullback-Leibler Divergence, and Mean
Absolute Error are not suitable for comparing distributions overordinal groups [11].
For each topic type, the GFR score is computed as follows.

3 TEST COLLECTION CONSTRUCTION
3.1 Target Corpus

3 https://research.nii.ac.jp/ntcir/tools/ntcireval-en.html

5 https://commoncrawl.org/blog/april-2021-crawl-archive-now-available

4 https://waseda.box.com/fairweb1pilotdata

6 https://github.com/chuzhumin98/Chuweb21D

We adopted Chuweb21D [5] as the target corpus. Chuweb21D
was constructed based on the Chuweb21 collection [5] throughde-duplication. Similar to Colossal Clean Crawled Corpus (c4) [9]and CC-News-En [7] datasets, the Chuweb21 collection was derived from the 2021-17 (April, 2021) block of the Common Crawl
dataset.5 The original 2021-17 block of Common Crawl data con!
tains about 3.1 billion web pages crawled from 10th to 23rd in April𝑙𝑀ÕÕ
GFR(𝐿)@𝑙 =Decay(𝐿, 𝑘) 𝑤 0 Utility(𝐿, 𝑘) +𝑤𝑚 DistrSim(𝐿, 𝑘) , 2021. After a series of filtering procedures including language detection, domain filtering and content length constraint, the cleaned
𝑚=1
𝑘=1
(2)
Chuweb21 collection contains 82.4 million web pages or 1.69 TiB
where 𝑙 is the document cutoff (we let 𝑙 = 20 for the official evalof compressed content.
uation), 𝑀 is the number of attribute sets considered (𝑀 = 2 for
The Chuweb21 collection was first used in the NTCIR-16 WWWM topics and R topics; 𝑀 = 1 for Y topics), 𝑤 0, . . . , 𝑤 𝑀 are the
4 task [19]. During the pooling set annotation stage, the task orweights for weighted averaging (we employ unweighted averagganizers found a fairly severe document duplication issue in theing and therefore 𝑤 0 = 𝑤 1 = 𝑤 2 = 1/3 for M and R topics, andChuweb21 collection, which reduces the efficiency of annotation
𝑤 0 = 𝑤 1 = 1/2 for Y topics).
resource usage and ranking system comparison. To solve this probThe Decay function in Eq. 2 is based on ERR (Expected Reciprocallem, Chu et al. [5] used the Simhash algorithm [8] to carry out massat ) isRank) [2, 10]. That is, the satisfaction probability at rank 𝑘 (𝑝 𝐿,𝑘sive de-duplication work on the Chuweb21 collection, and released
the de-duplicated Chuweb21D collection. The Chuweb21D collecdefined to be 3/4 for L2-relevant documents and 1/4 for L1-relevanttion contains two different releases obtained under different dedocuments, and the Decay is computed as:
duplication thresholds, Chuweb21D-60 and Chuweb21D-70, which𝑘−1Örepresent the scale of documents retained after de-duplication issatsat
Decay(𝐿, 𝑘) = 𝑝 𝐿,𝑘(1 − 𝑝 𝐿,𝑗) (𝑘 > 1)(3)
60% and 70% respectively. Specifically, we used Chuweb21D-60 as
𝑗=1
the target corpus, which contains about 49.8 million web pages andis now freely available online.6
sat .
and Decay(𝐿, 1) = 𝑝 𝐿,1
As for the Utility function in Eq. 2, we consider an ERR-based
3.2 Topics
function (Utility(𝐿, 𝑘) = 1/𝑘) as well as an iRBU (intentwise RankThefirst five authors of this paper constructed 45 test topics (15 R
Biased Utlity [20])-based function (Utility(𝐿, 𝑘) = 𝜙 𝑘 , with 𝜙 = 0.99
topics, 15 M topics, and 15 Y topics) based on our actual interests, so
throughout our evaluation). The ERR-based Decay value decreases
that they can serve as gold annotators (i.e., the annotators/assessors
rapidly as we go down the SERP (1,0.5, 0.33, 0.025, etc.), while theare the ones that have the actual information needs and therefore
iRBU-based one decreases very gradually (1, 0.99, 0.98, 0.97, etc.),
they get to decide what is relevant or not) [1, 18]. The topic set canand therefore the latter is probably more suitable for our task as
be found at https://waseda.box.com/fairweb1topics.
our topics are informational rather than navigational.
Eq. 2 generalises the Sakai/Robertson NCU (Normalised Cumula3.3 Submitted Runs
tive Utility) framework [16], which only had the Decay and Utility
We received 28 runs from five participant teams, including six runs
components. From this viewpoint, GFR represents the expectedfrom the organisers. Table 4 shows the system descriptions of eachuser experience under the ERR user model, where the experiencerun.
of each user group (i.e., those who abandon the SERP at a particular rank) is given as the weighted average of Utility and DistrSim
3.4 Entity Annotation
scores [17].
The entity annotation is done by utilising a assessment tool named
All scores were computed using NTCIREVAL version 230130.3
FAIRE (FAIRE stands for Annotation Interface for Relevant Entities).
Information on how to use this tool for the FairWeb-1 evaluation
This annotation interface is developed based on a tool named PLY,
can be found in the raw official results (See Section 4).
which is a relevance assessment tool used at the WWW tasks. At
FairWeb-1, the organisers who developed the test topics served as
2.3 Pilot Dataannotators, and each topic is assessed by the organiser who created
In February 2023, we released a pilot data set, which provides details
it. Each annotator was given a user account to access the interface.
on how our baseline runs are evaluated on a pilot topic set, which
Once the annotator logs in, the interface shows a list of assigned
contains one topic per each entity/.topic type (including a Twitter
topics and the annotation progress for each topic. Figure 1 is a
type that was later droppped).4
screenshot of FAIRE where a user just logs in. The annotator canchoose any topic she/he likes from the list to start annotation.

286

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 4: The SYSDESC field of each run. For brevity, the baseline run file names run.*.{query,description} are hereafter shown
as run.*.{Q,D}, respectively. See the participants’ papers for more details [3, 4, 6, 21].
Run
undisclosed-1
undisclosed-2
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5
THUIR-D-RR-5
THUIR-QD-RG-1
THUIR-QD-RG-2
THUIR-QD-RR-3
THUIR-QD-RR-4
UDinfo-D-RR-1
UDinfo-D-RR-3
UDinfo-D-RR-5
UDinfo-Q-RR-2
UDinfo-Q-RR-4
rmit_ir-D-RR-1
rmit_ir-D-RR-2
rmit_ir-D-RR-3
rmit_ir-D-RR-4
rmit_ir-Q-RR-5
run.bm25-depThre3-D
run.bm25-depThre3-Q
run.qld-depThre3-D
run.qld-depThre3-Q
run.qljm-depThre3-D
run.qljm-depThre3-Q

SYSDESC field
BM25 with query-field run
BM25 with description-field run
RSLFW Baseline using COIL with manual query operation
RSLFW Baseline + pm1 rerank algorithm
RSLFW Baseline + pm2 rerank algorithm
Official Baseline + pm2 rerank algorithm
Re-rank official baseline using Python library reranking
xQuAD + sparse relevance score
RRF, k = 60
Ultilize lightgbm to fuse 4 dense & 12 sparse relevance features
Ultilize prompts to calculate feature scores
PM2 + dense relevance score
re-rank run.bm25-D60-D_ver0313.txt leveraging gender and geo location embeddings
re-rank run.qld-D60-D_ver0313.txt leveraging gender and geo location embeddings
re-rank run.qljm-D60-D_ver0313.txt leveraging gender and geo location embeddings
re-rank run.bm25-D60-Q_ver0313.txt leveraging gender and geo location embeddings
re-rank run.qld-D60-Q_ver0313.txt leveraging gender and geo location embeddings
Linear combination of top 50 relevance and fairness with lambda= 0.9
PM2 with lambda= 0.9
PM2 on top 50 with lambda= 0.9
Linear combination of relevance and fairness with lambda= 0.9
Linear combination of top 50 relevance and fairness with lambda= 0.5
BM25 description-field run, with the default parameters of Anserini
BM25 query-field run, with the default parameters of Anserini
qld (query likelihood with Dirichlet smoothing) description-field run, with the default parameters of Anserini
qld (query likelihood with Dirichlet smoothing) query-field run, with the default parameters of Anserini
qljm (query likelihood with Jelinek-Mercer smoothing) description-field run, with the default parameters of Anserini
qljm (query likelihood with Jelinek-Mercer smoothing) query-field run, with the default parameters of Anserini

document relevance level 𝑔(𝑑) ∈ {0, 1, 2} of document 𝑑 is defined
as follows:0(𝐸 (𝑑) = ∅);𝑔(𝑑) =(4)max𝑒 ∈𝐸 (𝑑) 𝑟 (𝑒) (otherwise).

Figure 2 is a screenshot of FAIRE where the pool file for Topic
R005 has been loaded. The top area shows the current topic and description. The annotator can change to another topic by clicking theprev or the next button. By clicking the topic menu, the annotator
can also change to any other assigned topics from a dropdown list.
The left panel of the page shows the document pool for the current
topic. The user can annotate the pool in any order by clicking the
document in the left panel, and once the annotation of a document
is done, a coloured tag will be placed beside the document ID. In
the middle of the page, there is a document viewer and several text
fields for the annotator to read the document and fill the schema of
the top three relevant entities. If there is no relevant entity found in
the document, or the document is not shown properly (ex. FAIRE returns an error page, or the texts are unreadable), the annotator canclick the corresponding checkboxes below the text fields. Moreover,a comment field is shown at the bottom for the annotator to put any
remarks or commends regarding the document or the entities. Oncethe annotation of the document is done, the annotator can click
the save button to store the result to the database. The results areallowed to be updated if the annotator wants to make any change.

3.5

where E(d) is the set of relevant entities extracted from document 𝑑,and 𝑟 (𝑒) ∈ {1, 2} is the relevance level of a relevant entity 𝑒 ∈ 𝐸 (𝑑).
In other words, the relevance level of a document is simply definedas the max entity relevance level within the document.
As for the group membership probabilities of 𝑑, we conducted
two strategies to derive the probability vectors based on different
attribute sets. The first one is called hard group membership for
entities. This is for the scenario where an entity can be mapped toexactly one group of an attribute set. Let 𝐶 = {𝐶 1, ..., 𝐶 |𝐶 | } be an
attribute set, and 𝐹 (𝑒, 𝐶𝑖 ) be a flag that maps an entity 𝑒 to exactly
one group in 𝐶. For example, for a researcher entity 𝑒 whose bio says“he” (𝐶 = 𝐺𝐸𝑁 𝐷𝐸𝑅), 𝐹 (𝑒, 𝐶 1 ) = 1, 𝐹 (𝑒, 𝐶 2 ) = 𝐹 (𝑒, 𝐶 3 ) = 0. Then fora document 𝑑 with relevant entities 𝐸 (𝑑), the group membershipprobabilities of 𝑑 is defined as:(
1/|𝐶(𝐸 (𝑑) = ∅);Í|| 𝑒∈𝐸 (𝑑 ) 𝐹 (𝑒,𝐶𝑖 ) |𝑃 (𝑑, 𝐶𝑖 ) =(5)Í Í(otherwise).
|𝐹 (𝑒,𝐶 ) |𝑖

Deriving Document Relevance and Group
Membership𝑒∈𝐸 (𝑑 )𝑖

The second method to derive document group membership isnamed soft group membership, which is designed for the ORIGIN attribute set of movie entities. For a movie entity, it can bemapped to more than one group of the ORIGIN attribute set. Let

After the entity annotation is finished, the annotation results are
utilised to derive document relevance and group membership. The

287

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Figure 1: Screenshot of FAIRE, showing the assigned topics of an annotator

4.2

𝑂𝑅𝐼𝐺𝐼 𝑁 (𝑒) (⊆ 𝐶) be a set of geographical regions for a movie entity𝑒 ∈ 𝐸 (𝑑) (𝑚 = |𝑂𝑅𝐼𝐺𝐼 𝑁 (𝑒)|, 𝑚 ≧ 1). For example, if 𝑒’s countriesof origin is UK and China, then 𝑂𝑅𝐼𝐺𝐼 𝑁 (𝑒) = {𝐴𝑠𝑖𝑎, 𝐸𝑢𝑟𝑜𝑝𝑒}. For
movie entities, the group membership with respect to ORIGIN isdefined as follows:(
𝑃 (𝑑, 𝐶𝑖 ) =

1/|𝐶
Í|

Tables 6, 7 and 8 show the mean scores averaged across the 15 M
topics, with statistical test results for those involving group fairness.
As can be seen from the results, THUIR-QD-RR-3 is still the top
performer in terms of both relevance and group fairness. Notethat in Table 7, THUIR-QD-RR-3 outperforms run.qld-depThre3-D(an organiser run) statistically significantly in terms of GFNMD for
RATINGS.
Figures 3 and 4 visualise the mean scores of each system. From
Figure 3, we can see that the top three runs from team THUIR
substantially outperform the other runs in terms of mean iRBU.
Figure 4 shows the intersectional group fairness over the 15 M topics.
It can be seen that all the runs are generally “equally fair” to both
attributes, and there are no obvious bias towards either attribute.
The advantage of the three THUIR runs can also be observed from
Figure 4, they are the only runs have mean GF scores larger than
0.5 and forms the top cluster in the figure.
Figure 5 visalises the difficulty of each M topic in terms of scores
averaged over all runs. Topic M004 (Star Wars parody movies),
M014 (biographical movies), and M015 (fictional creature movies)
have much lower scores than the other M topics. The difficulty ofM004 probably comes from the limited correct answer to the query,while the reason why M014 and M015 have fewer relevant entities
remains further investigation.

(𝐸 (𝑑) = ∅);| 𝑒∈𝐸 (𝑑 ) 𝐺 (𝑒,𝐶𝑖 ) |Í Í| 𝑖 𝑒∈𝐸 (𝑑 ) 𝐺 (𝑒,𝐶𝑖 ) |(otherwise).

(6)

where

𝐺 (𝑒, 𝐶𝑖 ) =

4

1/𝑚
0

(𝐶𝑖 ∈ 𝑂𝑅𝐼𝐺𝐼 𝑁 (𝑒));(otherwise).

(7)

OFFICIAL RESULTS

The raw official result files and details can be found at https://waseda.box.com/ntcir17fairweb1officialpublic . Please go throughthe README file first.

4.1

Relevance and Group Fairness for Movie(M) Topics

Relevance: ERR and iRBU for the Full
Topic Set

Table 5 shows the run rankings according to Mean ERR and MeaniRBU over the entire topic set, respectively. The table indicatesclusters of runs in terms of statistical significance: for example,
THUIR-QD-RR-4 is the top ranker with respect to ERR, it outperforms statistically significantly the runs ranked from 18 to 28. And
THUIR-QD-RR-3 and THUIR-QD-RG-2 form the top cluster in termsof iRBU, by outperforming the runs ranked at 22 through 28.

288

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Figure 2: Screenshot of FAIRE, showing the annotation page

4.3

Relevance and Group Fairness for
Researcher (R) Topics

authors) is the most difficult topic amoung all R topics. The high ambguity of term “AIRS” probably results in retrieving many irrelevant
documents and leads to limited relevant hits.

Tables 9, 10 and 11 show the mean scores averaged across the 15 R
topics, with statistical test results for those involving group fairness.
The top 24 runs outperform the three zero-score runs, and there
are no other statistically significant differences amoung the runs.
Figures 6 and 7 visualise the mean scores of each system. It canbe observed that THUIR-QD-RG-2 and THUIR-QD-RG-1 are thebest two runs over the R topics, and they form the top cluster inFigure 7, which shows the intersectional group fairness over the R
topics. As can be seen, no runs have obvious bias towards either
attribute, while compared with what was observed in Figure 4, all
the runs tend to be have slightly higher GENDER GF scores than
HINDEX.
Figure 8 visalises the difficulty of each R topic in terms of scores
averaged over all runs. It can be observed that topic R004 (AIRS

4.4

Relevance and Group Fairness for YouTube(Y) Topics

Tables 12, 13 and 14 show the mean scores averaged across the
15 Y topics, with statistical test results for those involving groupfairness. In terms of GF and GFR scores, the top 17 runs outperfrom
the two zero-score runs, and there are no other statistical significant
differences observed.
Figures 9 visualises the mean scores of each system. The runs
from team THUIR can be found at the top ranks again over the Y
topics.
Figure 10 visalises the difficulty of each Y topic in terms of scores
averaged over all runs. From the figure, we can see that there are six
Y topics that have low scores, which is more than M and R topics.
The limited number of relevant documents in the corpus possibly
results in higher difficulty of these Y topics.

289

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 5: Mean ERR and iRBU scores for each run over the 45 topics. “>” means “stastistically significantly outperforms (according to a randomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05 [12]).” For example, in terms of ERR, THUIR-QD-RR-4
statistically significantly outperforms the runs ranked at 18 through 28.
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

4.5

Run
THUIR-QD-RR-4
THUIR-QD-RR-3
THUIR-QD-RG-2
RSLFW-Q-RR-5
UDinfo-Q-RR-2
rmit_ir-Q-RR-5
THUIR-QD-RG-1
run.qljm-depThre3-Q
RSLFW-Q-RR-4
UDinfo-Q-RR-4
run.bm25-depThre3-Q
rmit_ir-D-RR-4
UDinfo-D-RR-5
UDinfo-D-RR-3
rmit_ir-D-RR-1
UDinfo-D-RR-1
RSLFW-Q-MN-1
run.qld-depThre3-Q
THUIR-D-RR-5
run.qljm-depThre3-D
run.qld-depThre3-D
run.bm25-depThre3-D
rmit_ir-D-RR-3
RSLFW-Q-MN-2
rmit_ir-D-RR-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

Mean ERR
0.2139 (>18-28)
0.2074 (>20-28)
0.2021 (>22-28)
0.1847 (>26-28)
0.1788 (>26-28)
0.1685 (>26-28)
0.1542 (>27-28)
0.1495 (>27-28)
0.1485 (>27-28)
0.1465 (>27-28)
0.1390 (>27-28)
0.1379 (>27-28)
0.1343 (>27-28)
0.1310 (>27-28)
0.1306 (>27-28)
0.1306 (>27-28)
0.1245 (>27-28)
0.1226 (>27-28)
0.1218 (>27-28)
0.1152 (>27-28)
0.1126 (>27-28)
0.1113 (>27-28)
0.1084 (>27-28)
0.1067 (>27-28)
0.1029 (>27-28)
0.07700.00000.0000

Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

A Case Study: THUIR-QD-RG-2 vs.
run.qld-depThre3-D (for M topics)

Run
THUIR-QD-RG-2
THUIR-QD-RR-4
THUIR-QD-RR-3
UDinfo-Q-RR-2
RSLFW-Q-RR-5
UDinfo-Q-RR-4
THUIR-QD-RG-1
rmit_ir-Q-RR-5
THUIR-D-RR-5
RSLFW-Q-RR-4
UDinfo-D-RR-3
run.qljm-depThre3-Q
run.qld-depThre3-Q
rmit_ir-D-RR-1
UDinfo-D-RR-1
run.bm25-depThre3-Q
UDinfo-D-RR-5
rmit_ir-D-RR-4
rmit_ir-D-RR-3
run.qljm-depThre3-D
run.qld-depThre3-D
rmit_ir-D-RR-2
run.bm25-depThre3-D
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

Mean iRBU
0.5744 (>22-28)
0.5740 (>22-28)
0.5651 (>23-28)
0.4977 (>26-28)
0.4973 (>26-28)
0.4838 (>26-28)
0.4813 (>26-28)
0.4787 (>26-28)
0.4772 (>26-28)
0.4737 (>26-28)
0.4377 (>26-28)
0.4336 (>26-28)
0.4330 (>26-28)
0.4304 (>26-28)
0.4293 (>26-28)
0.4242 (>26-28)
0.4237 (>26-28)
0.4181 (>26-28)
0.4017 (>27-28)
0.3889 (>27-28)
0.3872 (>27-28)
0.3769 (>27-28)
0.3624 (>27-28)
0.3510 (>27-28)
0.3298 (>27-28)
0.2271 (>27-28)
0.00000.0000

the group membership for each L0 document (i.e., documents that
do not contain any relevant entities) is considered to be uniform
over the four groups of the RATINGS attribute set, while that foreach L1-relevant document is computed based on the number ofratings of each movie entity found within the document. Moreover,for example, in Table 15, the RNOD-based DistrSim for the 7th
document is computed as follows.
(1) By averaging over the group membership vectors from Ranks
1-7, the achieved distribution at Rank 7 is considered to be(0.2619, 0.3095, 0.2143, 0.2143).
(2) By comparing the above with the uniform gold distribution(0.25, 0.25, 0.25, 0.25), the RNOD-based DistrSim is computed
to be 0.9519.
Finally, GFRNOD (RATINGS) is obtained as the sum of 𝑃 ERR (𝑟 ) ∗DistrSim(𝑟 ), that is, 0.8867.
Similarly, the ORIGIN columns shows that the group membership for each L0 document is considered to be uniform over theeight groups of the ORIGIN attribute set, namely, Africa, America,
Antarctica, Asia, Caribbean, Europe, Middle East, Oceania, and that,not surprisingly, the majority of the relevant documents are biased
towards movies from America. In Table 15, the JSD-based DistrSim
for the 7th document is computed as follows.
(1) By averaging over the group membership vectors from Ranks
1-7, the achieved distribution at Rank 7 is considered to be

In Table 8, THUIR-QD-RG-2 is the top performer on average (for M
topics) in terms of both Mean GFJSD (ORIGIN) and Mean GFRNOD(RATINGS), whereas the worst performer in the same cluster (interms of all three measures) is the baseline run.qld-depThre3-D.
Also, in Table 5 where the entire topic set is used for computing
relevance-based measures, THUIR-QD-RG-2 is the top performer
on average in terms of Mean iRBU, and it statistically significantly
outperforms run.qld-depThre3-D which is ranked at 23. In thissection, we compare THUIR-QD-RG-2 and run.qld-depThre3-D
over the M topics to illustrate how the GFR framework works.
Figures 11-13 visualise the per-topic measure scores for THUIRQD-RG-2 and run.qld-depThre3-D. It can be observed that the iRBU,GFRNOD (for RATINGS), and GFJSD (for ORIGIN) scores are generally similar. Hereafter, we shal focus on Topic M012 (“cartoon
movies”), as THUIR-QD-RG-2 is about twice as effective as run.qlddepThre3-D in terms of all three measures.
Tables 15 and 16 show how group fairness scores are computedfor the SERPs from the two runs for Topic M012. First, it can beobserved that while the THUIR run manages to retrieve as manyas 12 L1-relevant documents, the baseline run retrieves only 2
L1-relevant documents; the ERR-based abandoning probabilities
𝑃ERR (𝑟 ) are also shown. Next, the RATINGS columns show that290

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 6: Mean ERR and iRBU scores for each run over the 15 M topics.
Run
THUIR-QD-RR-3
THUIR-QD-RR-4
rmit_ir-Q-RR-5
THUIR-QD-RG-2
run.qljm-depThre3-Q
RSLFW-Q-RR-5
UDinfo-Q-RR-2
UDinfo-Q-RR-4
RSLFW-Q-MN-1
rmit_ir-D-RR-4
rmit_ir-D-RR-1
run.bm25-depThre3-Q
run.qld-depThre3-Q
RSLFW-Q-RR-4
THUIR-QD-RG-1
UDinfo-D-RR-1
run.bm25-depThre3-D
UDinfo-D-RR-5
RSLFW-Q-MN-3
RSLFW-Q-MN-2
run.qljm-depThre3-D
rmit_ir-D-RR-2
rmit_ir-D-RR-3
THUIR-D-RR-5
UDinfo-D-RR-3
run.qld-depThre3-D
undisclosed-1
undisclosed-2

Mean ERR
0.26530.25180.24340.22800.21140.20440.2017
0.19570.18930.18870.17220.17120.16530.16200.16080.15820.15640.14990.14890.14890.14780.14720.14660.12230.12220.11870.00000.0000Run
THUIR-QD-RR-3
THUIR-QD-RG-2
THUIR-QD-RR-4
run.qljm-depThre3-Q
rmit_ir-Q-RR-5
RSLFW-Q-RR-5
UDinfo-Q-RR-2
UDinfo-Q-RR-4
RSLFW-Q-RR-4
THUIR-D-RR-5
rmit_ir-D-RR-4
UDinfo-D-RR-5
run.bm25-depThre3-Q
run.qld-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-1
rmit_ir-D-RR-3
UDinfo-D-RR-1
rmit_ir-D-RR-2
THUIR-QD-RG-1
run.bm25-depThre3-D
RSLFW-Q-MN-1
RSLFW-Q-MN-3
RSLFW-Q-MN-2
UDinfo-D-RR-3
run.qld-depThre3-D
undisclosed-1
undisclosed-2(0.1071, 0.1786, 0.1071, 0.1786, 0.1071, 0.1071, 0.1071, 0.1071). Notethat the probabilities for America and Asia are slightly higher
than the rest due to the group membership vector of the 7th
document.
(2) By comparing the above with the gold distribution, namely,(0.2447, 0.1118, 0.0084, 0.1540, 0.0992, 0.2004, 0.08020.1013), theJSD-based DistrSim is computed to be 0.9259.
Finally, GFJSD (ORIGIN) is obtained as the sum of 𝑃ERR (𝑟 )∗DistrSim(𝑟 ),that is, 0.8630.

291

Mean iRBU
0.72300.69230.68590.6026
0.58190.56740.56680.55820.54630.5316
0.52390.5136
0.50350.49580.4883
0.48620.47410.45850.45790.44000.43370.42680.42500.42500.40690.37280.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 7: Mean group fairness scores for each run over the 15 M topics. “>” means “stastistically significantly outperforms(according to a randomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05.”
Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RG-2
THUIR-QD-RR-3
THUIR-QD-RR-4
THUIR-D-RR-5
RSLFW-Q-RR-4
run.qljm-depThre3-Q
UDinfo-Q-RR-4
UDinfo-D-RR-5
UDinfo-Q-RR-2
RSLFW-Q-RR-5
run.qld-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-4
rmit_ir-Q-RR-5
run.bm25-depThre3-Q
rmit_ir-D-RR-3
rmit_ir-D-RR-1
rmit_ir-D-RR-2
UDinfo-D-RR-1
RSLFW-Q-MN-3
RSLFW-Q-MN-2
UDinfo-D-RR-3
run.bm25-depThre3-D
THUIR-QD-RG-1
RSLFW-Q-MN-1
run.qld-depThre3-D
undisclosed-1
undisclosed-2

Mean GFJSD(ORIGIN)
0.5684 (>27-28)
0.5391 (>27-28)
0.5332 (>27-28)
0.4900 (>27-28)
0.4768 (>27-28)
0.4716 (>27-28)
0.4601 (>27-28)
0.4543 (>27-28)
0.4493 (>27-28)
0.4479 (>27-28)
0.4275 (>27-28)
0.4273 (>27-28)
0.4211 (>27-28)
0.4177 (>27-28)
0.4135 (>27-28)
0.3989 (>27-28)
0.3842 (>27-28)
0.3772 (>27-28)
0.3672 (>27-28)
0.3514 (>27-28)
0.3514 (>27-28)
0.3476 (>27-28)
0.3401 (>27-28)
0.3395 (>27-28)
0.3179 (>27-28)
0.3122 (>27-28)
0.00000.0000

Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RR-3
THUIR-QD-RG-2
THUIR-QD-RR-4
run.qljm-depThre3-Q
THUIR-D-RR-5
RSLFW-Q-RR-4
UDinfo-Q-RR-4
UDinfo-Q-RR-2
RSLFW-Q-RR-5
rmit_ir-Q-RR-5
UDinfo-D-RR-5
rmit_ir-D-RR-4
run.qld-depThre3-Q
run.bm25-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-3
rmit_ir-D-RR-1
rmit_ir-D-RR-2
UDinfo-D-RR-1
THUIR-QD-RG-1
run.bm25-depThre3-D
UDinfo-D-RR-3
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qld-depThre3-D
undisclosed-1
undisclosed-2

292

Mean GFNMD(RATINGS)
0.6433 (>26-28)
0.6330 (>27-28)
0.6118 (>27-28)
0.5462 (>27-28)
0.5307 (>27-28)
0.5169 (>27-28)
0.5161 (>27-28)
0.5132 (>27-28)
0.5124 (>27-28)
0.5043 (>27-28)
0.4888 (>27-28)
0.4784 (>27-28)
0.4668 (>27-28)
0.4623 (>27-28)
0.4606 (>27-28)
0.4529 (>27-28)0.4502 (>27-28)
0.4309 (>27-28)
0.4279 (>27-28)
0.4025 (>27-28)
0.3993 (>27-28)
0.3876 (>27-28)
0.3847 (>27-28)
0.3847 (>27-28)
0.3718 (>27-28)
0.3507 (>27-28)
0.00000.0000

Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RG-2
THUIR-QD-RR-3
THUIR-QD-RR-4
THUIR-D-RR-5
run.qljm-depThre3-Q
RSLFW-Q-RR-4
UDinfo-Q-RR-4
UDinfo-Q-RR-2
RSLFW-Q-RR-5
UDinfo-D-RR-5
rmit_ir-Q-RR-5
run.qld-depThre3-Q
run.bm25-depThre3-Q
rmit_ir-D-RR-4
rmit_ir-D-RR-3
run.qljm-depThre3-D
rmit_ir-D-RR-1
rmit_ir-D-RR-2
UDinfo-D-RR-1
THUIR-QD-RG-1
run.bm25-depThre3-D
UDinfo-D-RR-3
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qld-depThre3-D
undisclosed-1
undisclosed-2

Mean GFRNOD(RATINGS)
0.5788 (>27-28)
0.5683 (>27-28)
0.5435 (>27-28)
0.4983 (>27-28)
0.4871 (>27-28)
0.4758 (>27-28)
0.4750 (>27-28)
0.4706 (>27-28)
0.4693 (>27-28)
0.4488 (>27-28)
0.4480 (>27-28)
0.4351 (>27-28)
0.4283 (>27-28)
0.4281 (>27-28)
0.4234 (>27-28)
0.4211 (>27-28)
0.4062 (>27-28)
0.4035 (>27-28)
0.3913 (>27-28)
0.3684 (>27-28)
0.3630 (>27-28)
0.3569 (>27-28)
0.3503 (>27-28)
0.3503 (>27-28)
0.3296 (>27-28)
0.3208 (>27-28)
0.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 8: Mean GFR score for each run over the 15 M topics: relevance based on iRBU; group fairness for RATINGS basedon RNOD. (Group fairness for ORIGIN is based on JSD.) “>” means “stastistically significantly outperforms (according to arandomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05.”
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Run
THUIR-QD-RG-2
THUIR-QD-RR-3
THUIR-QD-RR-4
run.qljm-depThre3-Q
THUIR-D-RR-5
RSLFW-Q-RR-4
UDinfo-Q-RR-4
UDinfo-Q-RR-2
RSLFW-Q-RR-5
rmit_ir-Q-RR-5
UDinfo-D-RR-5
rmit_ir-D-RR-4
run.qld-depThre3-Q
run.bm25-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-3
rmit_ir-D-RR-1
rmit_ir-D-RR-2
UDinfo-D-RR-1
THUIR-QD-RG-1
run.bm25-depThre3-D
RSLFW-Q-MN-3
RSLFW-Q-MN-2
UDinfo-D-RR-3
RSLFW-Q-MN-1
run.qld-depThre3-D
undisclosed-1
undisclosed-2

293

Mean GFR
0.6132 (> 27-28)
0.6101 (> 27-28)
0.5875 (> 27-28)
0.5205 (> 27-28)
0.5066 (> 27-28)
0.4996 (> 27-28)
0.4977 (> 27-28)
0.4956 (> 27-28)
0.4949 (> 27-28)
0.4825 (> 27-28)
0.4722 (> 27-28)
0.4577 (> 27-28)
0.4528 (> 27-28)
0.4484 (> 27-28)
0.4456 (> 27-28)
0.4321 (> 27-28)
0.4255 (> 27-28)
0.4129 (> 27-28)
0.4057 (> 27-28)
0.3827 (> 27-28)
0.3789 (> 27-28)
0.3756 (> 27-28)
0.3756 (> 27-28)
0.3705 (> 27-28)
0.3581 (> 27-28)
0.3353 (> 27-28)
0.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

0.8

0.7
0.6
0.5
0.4
0.3
0.2
0.1

RNOD

Mean iRBU

Mean GF(RATINGS)

JSD

UDinfo-D-RR-3

run.qld-depThre3-D

RSLFW-Q-MN-3

RSLFW-Q-MN-2

RSLFW-Q-MN-1

run.bm25-depThre3-D

rmit_ir-D-RR-2

Mean GF

THUIR-QD-RG-1

rmit_ir-D-RR-3

UDinfo-D-RR-1

rmit_ir-D-RR-1

run.qld-depThre3-Q

run.qljm-depThre3-D

run.bm25-depThre3-Q

rmit_ir-D-RR-4

UDinfo-D-RR-5

THUIR-D-RR-5

UDinfo-Q-RR-4

RSLFW-Q-RR-4

UDinfo-Q-RR-2

rmit_ir-Q-RR-5

RSLFW-Q-RR-5

THUIR-QD-RR-4

run.qljm-depThre3-Q

THUIR-QD-RR-3

THUIR-QD-RG-2

0(ORIGIN)

Figure 3: Visualisation of mean scores over the 15 M topics.

0.55

0.5

Mean GF JSD (ORIGIN)

0.6

0.45

0.4

0.35

0.3

0.25
Mean GF RNOD (RATINGS)

0.2
0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

Figure 4: Intersectional group fairness: mean scores over the 15 M topics.

294

0.6

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

0.8
0.7
0.6
0.5

0.4
0.3
0.2
0.1
0
M001 M002 M003 M004 M005 M006 M007 M008 M009 M010 M011 M012 M013 M014 M015
Average iRBU

RNOD

Average GF(RATINGS)

Average GF

JSD(ORIGIN)

Figure 5: Visualisation of topic difficulty in terms of average score across all runs (M topics).
Table 9: Mean ERR and iRBU scores for each run over the 15 R topics
Run
THUIR-QD-RG-2
THUIR-QD-RR-4
THUIR-QD-RR-3
RSLFW-Q-RR-5
run.qljm-depThre3-Q
run.bm25-depThre3-Q
UDinfo-Q-RR-2
THUIR-QD-RG-1
run.qld-depThre3-D
UDinfo-D-RR-3
run.qld-depThre3-Q
UDinfo-Q-RR-4
UDinfo-D-RR-1
run.bm25-depThre3-D
RSLFW-Q-RR-4
run.qljm-depThre3-D
UDinfo-D-RR-5
THUIR-D-RR-5
rmit_ir-D-RR-4
rmit_ir-Q-RR-5
rmit_ir-D-RR-1
rmit_ir-D-RR-3
RSLFW-Q-MN-1
RSLFW-Q-MN-2
rmit_ir-D-RR-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

Mean ERR
0.26380.24600.2276
0.21310.2104
0.19890.19890.19180.17490.17310.15670.15500.15320.15090.14780.14590.14320.14210.13740.13280.13090.10360.10210.08900.08670.00000.00000.0000Run
THUIR-QD-RG-2
THUIR-QD-RG-1
THUIR-QD-RR-4
THUIR-QD-RR-3
run.qld-depThre3-D
UDinfo-D-RR-3
run.qld-depThre3-Q
run.bm25-depThre3-Q
UDinfo-Q-RR-2
RSLFW-Q-RR-5
UDinfo-Q-RR-4
THUIR-D-RR-5
UDinfo-D-RR-1
RSLFW-Q-RR-4
run.qljm-depThre3-Q
rmit_ir-Q-RR-5
rmit_ir-D-RR-1
run.bm25-depThre3-D
run.qljm-depThre3-D
rmit_ir-D-RR-3
UDinfo-D-RR-5
rmit_ir-D-RR-4
RSLFW-Q-MN-1
rmit_ir-D-RR-2
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

295

Mean iRBU
0.65600.60130.59570.58040.56950.55820.55180.54890.54890.54880.53670.53510.50550.49740.4971
0.48800.48160.48010.43610.41330.41000.40630.37000.35510.30840.00000.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 10: Mean group fairness scores for each run over the 15 R topics. “>” means “stastistically significantly outperforms(according to a randomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05.”
Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RG-2
THUIR-QD-RG-1
run.qld-depThre3-D
UDinfo-D-RR-3
run.qld-depThre3-Q
THUIR-D-RR-5
UDinfo-Q-RR-4
run.bm25-depThre3-Q
UDinfo-Q-RR-2
THUIR-QD-RR-4
THUIR-QD-RR-3
RSLFW-Q-RR-5
UDinfo-D-RR-1
rmit_ir-Q-RR-5
RSLFW-Q-RR-4
rmit_ir-D-RR-1
run.bm25-depThre3-D
run.qljm-depThre3-Q
rmit_ir-D-RR-3
run.qljm-depThre3-D
rmit_ir-D-RR-4
UDinfo-D-RR-5
RSLFW-Q-MN-1
rmit_ir-D-RR-2
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

Mean GFJSD(GENDER)
0.5831 (>26-28)
0.5823 (>26-28)
0.5497 (>26-28)
0.5374 (>26-28)
0.5356 (>26-28)
0.5351 (>26-28)
0.5190 (>26-28)
0.5096 (>26-28)
0.5096 (>26-28)
0.5086 (>26-28)
0.4987 (>26-28)
0.4986 (>26-28)
0.4985 (>26-28)
0.4927 (>26-28)
0.4886 (>26-28)
0.4819 (>26-28)
0.4694 (>26-28)
0.4315 (>26-28)
0.4125 (>26-28)
0.4120 (>26-28)
0.3861 (>26-28)
0.3829 (>26-28)
0.3771 (>26-28)
0.3572 (>26-28)
0.30800.00000.00000.0000

Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RG-2
THUIR-QD-RG-1
run.qld-depThre3-D
THUIR-QD-RR-3
UDinfo-D-RR-3
THUIR-QD-RR-4
run.qld-depThre3-Q
THUIR-D-RR-5
UDinfo-Q-RR-4
run.bm25-depThre3-Q
UDinfo-Q-RR-2
RSLFW-Q-RR-5
RSLFW-Q-RR-4
rmit_ir-Q-RR-5
rmit_ir-D-RR-1
UDinfo-D-RR-1
run.bm25-depThre3-D
run.qljm-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-3
rmit_ir-D-RR-4
UDinfo-D-RR-5
RSLFW-Q-MN-1
rmit_ir-D-RR-2
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

296

Mean GFNMD(HINDEX)
0.5841 (>26-28)
0.5569 (>26-28)
0.5306 (>26-28)
0.5247 (>26-28)
0.5195 (>26-28)
0.5164 (>26-28)
0.5152 (>26-28)
0.5080 (>26-28)
0.4994 (>26-28)
0.4977 (>26-28)
0.4977 (>26-28)
0.4942 (>26-28)
0.4815 (>26-28)
0.4778 (>26-28)
0.4751 (>26-28)
0.4682 (>26-28)
0.4400 (>26-28)
0.4362 (>26-28)
0.4038 (>26-28)
0.4006 (>26-28)
0.3858 (>26-28)
0.3765 (>26-28)
0.3563 (>26-28)
0.3420 (>26-28)
0.30400.00000.00000.0000

Rank

Run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

THUIR-QD-RG-2
THUIR-QD-RG-1
run.qld-depThre3-D
THUIR-QD-RR-3
UDinfo-D-RR-3
THUIR-D-RR-5
run.qld-depThre3-Q
THUIR-QD-RR-4
UDinfo-Q-RR-4
run.bm25-depThre3-Q
UDinfo-Q-RR-2
RSLFW-Q-RR-4
RSLFW-Q-RR-5
rmit_ir-Q-RR-5
rmit_ir-D-RR-1
UDinfo-D-RR-1
run.bm25-depThre3-D
run.qljm-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-3
rmit_ir-D-RR-4
UDinfo-D-RR-5
RSLFW-Q-MN-1
rmit_ir-D-RR-2
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

Mean GFRNOD(HINDEX)
0.5352 (>26-28)
0.5257 (>26-28)
0.4975 (>26-28)
0.4875 (>26-28)
0.4866 (>26-28)
0.4841 (>26-28)
0.4807 (>26-28)
0.4720 (>26-28)
0.4650 (>26-28)
0.4605 (>26-28)
0.4605 (>26-28)
0.4562 (>26-28)
0.4556 (>26-28)
0.4530 (>26-28)
0.4509 (>26-28)
0.4434 (>26-28)
0.4155 (>26-28)
0.3999 (>26-28)
0.3824 (>26-28)
0.3815 (>26-28)
0.3613 (>26-28)
0.3554 (>26-28)
0.3395 (>26-28)
0.3255 (>26-28)
0.29160.00000.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 11: Mean GFR score for each run over the 15 R topics: relevance based on iRBU; group fairness for HINDEX basedon RNOD. (Group fairness for GENDER is based on JSD.) “>” means “stastistically significantly outperforms (according to arandomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05.”
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Run
THUIR-QD-RG-2
THUIR-QD-RG-1
run.qld-depThre3-D
UDinfo-D-RR-3
THUIR-QD-RR-4
run.qld-depThre3-Q
THUIR-QD-RR-3
THUIR-D-RR-5
UDinfo-Q-RR-4
run.bm25-depThre3-Q
UDinfo-Q-RR-2
RSLFW-Q-RR-5
UDinfo-D-RR-1
RSLFW-Q-RR-4
rmit_ir-Q-RR-5
rmit_ir-D-RR-1
run.bm25-depThre3-D
run.qljm-depThre3-Q
run.qljm-depThre3-D
rmit_ir-D-RR-3
rmit_ir-D-RR-4
UDinfo-D-RR-5
RSLFW-Q-MN-1
rmit_ir-D-RR-2
RSLFW-Q-MN-2
RSLFW-Q-MN-3
undisclosed-1
undisclosed-2

297

Mean GFR
0.5914 (>26-28)
0.5698 (>26-28)
0.5389 (>26-28)
0.5274 (>26-28)
0.5254 (>26-28)
0.5227 (>26-28)
0.5222 (>26-28)
0.5181 (>26-28)
0.5069 (>26-28)
0.5064 (>26-28)
0.5064 (>26-28)
0.5010 (>26-28)
0.4824 (>26-28)
0.4807 (>26-28)
0.4779 (>26-28)
0.4714 (>26-28)
0.4550 (>26-28)
0.4428 (>26-28)
0.4101 (>26-28)
0.4025 (>26-28)
0.3846 (>26-28)
0.3828 (>26-28)
0.3622 (>26-28)
0.3459 (>26-28)
0.30270.00000.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

0.8

0.7
0.6
0.5
0.4
0.3
0.2
0.1

Mean GF RNOD (HINDEX)

Mean GF JSD

rmit_ir-D-RR-2

RSLFW-Q-MN-2

RSLFW-Q-MN-1

rmit_ir-D-RR-4

rmit_ir-D-RR-3

UDinfo-D-RR-5

run.qljm-depThre3-D

rmit_ir-D-RR-1

run.bm25-depThre3-D

rmit_ir-Q-RR-5

run.qljm-depThre3-Q

UDinfo-D-RR-1

RSLFW-Q-RR-4

THUIR-D-RR-5

UDinfo-Q-RR-4

RSLFW-Q-RR-5

UDinfo-Q-RR-2

run.bm25-depThre3-Q

UDinfo-D-RR-3

Mean iRBU

run.qld-depThre3-Q

THUIR-QD-RR-3

run.qld-depThre3-D

THUIR-QD-RR-4

THUIR-QD-RG-1

THUIR-QD-RG-2

0(GENDER)

Figure 6: Visualisation of mean scores over the 15 R topics.

0.55

0.5

Mean GF JSD (GENDER)

0.6

0.45

0.4

0.35

0.3

0.25
Mean GF RNOD (HINDEX)

0.2
0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

Figure 7: Intersectional group fairness: mean scores over the 15 R topics.

298

0.6

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

0.8
0.7
0.6
0.5

0.4
0.3
0.2
0.1
0
R001 R002 R003 R004 R005 R006 R007 R008 R009 R010 R011 R012 R013 R014 R015
Average iRBU

Average GF RNOD (HINDEX)

Average GF JSD (GENDER)

Figure 8: Visualisation of topic difficulty in terms of average score across all runs (R topics).
Table 12: Mean ERR and iRBU scores for each run over the 15 Y topics
Run
THUIR-QD-RR-4
RSLFW-Q-RR-5
UDinfo-Q-RR-2
RSLFW-Q-RR-4
rmit_ir-Q-RR-5
THUIR-QD-RR-3
THUIR-QD-RG-2
THUIR-QD-RG-1
UDinfo-D-RR-5
THUIR-D-RR-5
UDinfo-D-RR-3
rmit_ir-D-RR-1
UDinfo-Q-RR-4
rmit_ir-D-RR-4
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
UDinfo-D-RR-1
rmit_ir-D-RR-3
rmit_ir-D-RR-2
run.qljm-depThre3-D
run.bm25-depThre3-Q
run.qld-depThre3-Q
run.qld-depThre3-D
run.qljm-depThre3-Q
run.bm25-depThre3-D
undisclosed-1
undisclosed-2

Mean ERR
0.14380.13650.13570.13570.12940.12930.1144
0.10990.10980.10090.09780.08880.08870.08770.08220.08220.08220.08040.07490.07490.05200.04710.04590.04420.0266
0.02660.00000.0000Run
THUIR-QD-RR-4
THUIR-QD-RG-1
THUIR-QD-RR-3
UDinfo-Q-RR-2
RSLFW-Q-RR-4
RSLFW-Q-RR-5
THUIR-QD-RG-2
rmit_ir-Q-RR-5
THUIR-D-RR-5
UDinfo-Q-RR-4
UDinfo-D-RR-3
UDinfo-D-RR-5
rmit_ir-D-RR-4
UDinfo-D-RR-1
rmit_ir-D-RR-1
rmit_ir-D-RR-3
rmit_ir-D-RR-2
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qld-depThre3-Q
run.qljm-depThre3-D
run.bm25-depThre3-Q
run.qld-depThre3-D
run.qljm-depThre3-Q
run.bm25-depThre3-D
undisclosed-1
undisclosed-2

299

Mean iRBU
0.44040.4026
0.39190.37750.37750.37570.37490.36620.36490.35640.34790.34760.32400.32390.32330.31770.31770.25620.25620.25620.25140.24240.22020.21940.20100.17350.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 13: Mean group fairness scores for each run over the 15 Y topics. “>” means “stastistically significantly outperforms(according to a randomised Tukey HSD test with 𝐵 = 5, 000 trials, 𝛼 = 0.05.”
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Run
THUIR-QD-RR-4
THUIR-QD-RG-1
THUIR-QD-RR-3
THUIR-D-RR-5
THUIR-QD-RG-2
UDinfo-Q-RR-2
RSLFW-Q-RR-4
UDinfo-Q-RR-4
RSLFW-Q-RR-5
UDinfo-D-RR-5
UDinfo-D-RR-3
rmit_ir-Q-RR-5
UDinfo-D-RR-1
rmit_ir-D-RR-3
rmit_ir-D-RR-2
rmit_ir-D-RR-4
rmit_ir-D-RR-1
run.qld-depThre3-Q
run.qljm-depThre3-D
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qld-depThre3-D
run.bm25-depThre3-Q
run.qljm-depThre3-Q
run.bm25-depThre3-D
undisclosed-1
undisclosed-2

Mean GFNMD (SUBSCS)
0.4112 (>27-28)
0.3830 (>27-28)
0.3601 (>27-28)
0.3550 (>27-28)
0.3423 (>27-28)
0.3315 (>27-28)
0.3315 (>27-28)
0.3309 (>27-28)
0.3292 (>27-28)
0.3279 (>27-28)
0.3228 (>27-28)
0.3169 (>27-28)
0.3157 (>27-28)
0.3146 (>27-28)
0.3146 (>27-28)
0.3100 (>27-28)
0.3084 (>27-28)
0.2451
0.24250.23390.23390.23390.21550.21120.20710.17770.00000.0000

Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

300

Run
THUIR-QD-RR-4
THUIR-QD-RG-1
THUIR-D-RR-5
THUIR-QD-RR-3
UDinfo-Q-RR-4
THUIR-QD-RG-2
UDinfo-D-RR-3
UDinfo-D-RR-5
UDinfo-Q-RR-2
RSLFW-Q-RR-4
RSLFW-Q-RR-5
rmit_ir-D-RR-3
rmit_ir-D-RR-2
UDinfo-D-RR-1
rmit_ir-D-RR-4
rmit_ir-D-RR-1
rmit_ir-Q-RR-5
run.qld-depThre3-Q
run.qljm-depThre3-D
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qld-depThre3-D
run.bm25-depThre3-Q
run.qljm-depThre3-Q
run.bm25-depThre3-D
undisclosed-1
undisclosed-2

Mean GFRNOD (SUBSCS)
0.3809 (>27-28)
0.3638 (>27-28)
0.3396 (>27-28)
0.3297 (>27-28)
0.3157 (>27-28)
0.3141 (>27-28)
0.3091 (>27-28)
0.3083 (>27-28)
0.3081 (>27-28)
0.3081 (>27-28)
0.3058 (>27-28)
0.3025 (>27-28)
0.3025 (>27-28)
0.3017 (>27-28)
0.2945 (>27-28)
0.2928 (>27-28)
0.2915 (>27-28)
0.23910.23290.22010.22010.22010.21000.20390.20380.17310.00000.0000NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 14: Mean GFR score for each run over the 15 Y topics: relevance based on iRBU; group fairness for SUBSCS based on
RNOD. “>” means “stastistically significantly outperforms (according to a randomised Tukey HSD test with 𝐵 = 5, 000 trials,
𝛼 = 0.05.”
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Run
THUIR-QD-RR-4
THUIR-QD-RG-1
THUIR-QD-RR-3
THUIR-D-RR-5
THUIR-QD-RG-2
UDinfo-Q-RR-2
RSLFW-Q-RR-4
RSLFW-Q-RR-5
UDinfo-Q-RR-4
rmit_ir-Q-RR-5
UDinfo-D-RR-3
UDinfo-D-RR-5
UDinfo-D-RR-1
rmit_ir-D-RR-3
rmit_ir-D-RR-2
rmit_ir-D-RR-4
rmit_ir-D-RR-1
run.qld-depThre3-Q
RSLFW-Q-MN-3
RSLFW-Q-MN-2
RSLFW-Q-MN-1
run.qljm-depThre3-D
run.qld-depThre3-D
run.bm25-depThre3-Q
run.qljm-depThre3-Q
run.bm25-depThre3-D
undisclosed-1
undisclosed-2

Mean GFR
0.4107 (>27-28)
0.3832 (>27-28)
0.3608 (>27-28)
0.3523 (>27-28)
0.3445 (>27-28)
0.3428 (>27-28)
0.3428 (>27-28)
0.3408 (>27-28)
0.3361 (>27-28)
0.3288 (>27-28)
0.3285 (>27-28)
0.3280 (>27-28)
0.3128 (>27-28)
0.3101 (>27-28)
0.3101 (>27-28)
0.3092 (>27-28)
0.3080 (>27-28)
0.24530.23810.23810.2381
0.23770.21470.21210.20240.17330.00000.0000

Table 15: Evaluating the SERP of THUIR-QD-RG-2 for Topic M012.
Rank𝑟

Relevancelevel

𝑃 ERR (𝑟 )

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

L0
L0
L0
L0
L0
L0
L1
L0
L1
L1
L1
L1
L1
L0
L1
L1
L1
L1
L1
L1

0
0
0
0
0
0
0.250000.18750.14060.10550.07910.059300.04450.03340.02500.01880.01410.0106

RATINGS
Membership
DistrSim(𝑟 )(RNOD-based)
0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.33/0.67/0/0
0.95190.25 per group0/0.67/0.33/0
0.93150/0.33/0.67/0
0.91820/1/0/0
0.88330/0/1/0
0.88050/0.33/0.67/0
0.86660.25 per group1/0/0/0
0.8963
1/0/0/00.9005
0/0.50/0.50/0
0.8926
1/0/0/0
0.88950/0/1/0
0.8846
1/0/0/0
0.8783
GFRNOD (RATINGS)
0.8867301

ORIGIN
Membership
0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0/0.5/0/0.5/0/0/0/0
0.125 per group0/0.67/0/0/0/0.33/0/0
0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
0/0.78/0/0.11/0/0.11/0/0
0/0.83/0/0.17/0/0/0/0
0.125 per group0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
0/1/0/0/0/0/0/0
GFJSD (ORIGIN)

DistrSim(𝑟 )(JSD-based)

0.92590.92490.90310.87990.86680.85110.84270.82530.80890.79350.77890.76530.8630NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

0.8

0.7
0.6
0.5
0.4
0.3
0.2
0.1

Mean iRBU

run.bm25-depThre3-D

run.qld-depThre3-D

run.qljm-depThre3-Q

run.qljm-depThre3-D

run.bm25-depThre3-Q

RSLFW-Q-MN-3

run.qld-depThre3-Q

RSLFW-Q-MN-2

rmit_ir-D-RR-3

RSLFW-Q-MN-1

rmit_ir-D-RR-2

rmit_ir-D-RR-1

rmit_ir-D-RR-4

UDinfo-D-RR-1

UDinfo-D-RR-5

UDinfo-D-RR-3

UDinfo-Q-RR-4

rmit_ir-Q-RR-5

THUIR-D-RR-5

THUIR-QD-RG-2

UDinfo-Q-RR-2

RSLFW-Q-RR-5

RSLFW-Q-RR-4

THUIR-QD-RR-3

THUIR-QD-RR-4

THUIR-QD-RG-1

0

Mean GF RNOD (SUBSCS)

Figure 9: Visualisation of mean scores over the 15 Y topics.
0.8
0.7
0.6
0.5

0.4
0.3
0.2
0.1
0
Y001 Y002 Y003 Y004 Y005 Y006 Y007 Y008 Y009 Y010 Y011 Y012 Y013 Y014 Y015
Average iRBU

Average GF RNOD (SUBSCS)

Figure 10: Visualisation of topic difficulty in terms of average score across all runs (Y topics).

302

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

THUIR-QD-RG-2

run.qld-depThre3-D

Figure 11: THUIR-QD-RG-2 vs. run.qld-depThre3-D over the M topics (iRBU).
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

THUIR-QD-RG-2

run.qld-depThre3-D

Figure 12: THUIR-QD-RG-2 vs. run.qld-depThre3-D over the M topics (GFRNOD for RATINGS).
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

THUIR-QD-RG-2

run.qld-depThre3-D

Figure 13: THUIR-QD-RG-2 vs. run.qld-depThre3-D over the M topics (GFJSD for ORIGIN).

303

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 16: Evaluating the SERP of run.qld-depThre3-D for Topic M012.
Rank𝑟

Relevancelevel

𝑃 ERR (𝑟 )

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

L0
L0
L0
L0
L0
L0
L0
L0
L0
L0
L0
L0
L0
L1
L0
L0
L0
L1
L0
L0

0
0
0
0
0
0
0
0
0
0
0
0
0
0.25000000.187500

RATINGS
Membership
DistrSim(𝑟 )(RNOD-based)
0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0.25 per group0/0/1/0
0.96280.25 per group0.25 per group0.25 per group0.33/0.67/0/0
0.97330.25 per group0.25 per group
GFRNOD (RATINGS)
0.4232304

ORIGIN
Membership
0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0.125 per group0/0.78/0/0.11/0/0.11/0/0
0.125 per group0.125 per group0.125 per group0/0.5/0/0.5/0/0/0/0
0.125 per group0.125 per group
GFJSD (ORIGIN)

DistrSim(𝑟 )(JSD-based)

0.92760.92730.4058NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

5

CONCLUSIONS[12] Tetsuya Sakai. 2018. Laboratory Experiments in Information Retrieval: Sample
Sizes, Effect Sizes, and Statistical Power. Springer. https://link.springer.com/book/10.1007/978-981-13-1199-4[13] Tetsuya Sakai. 2020. Evaluating Evaluation Measures for Ordinal Classification
and Ordinal Quantification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long Papers) (Online). Association for
Computational Linguistics, 2759–2769.
[14] Tetsuya Sakai. 2021. A Closer Look at Evaluation Measures for Ordinal Quantification. In Proceedings of the CIKM 2021 Workshops co-located with 30th ACM
International Conference on Information and Knowledge Management (CIKM 2021).
https://ceur-ws.org/Vol-3052/paper21.pdf[15] Tetsuya Sakai. 2023. Fairness-based Evaluation of Conversational Search: A Pilot
Study. In Proceedings of EVIA 2023.
[16] Tetsuya Sakai and Noriko Kando. 2008. Are Popular Documents More Likely to
be Relevant? A Dive into the ACLIA IR4QA Pools. In Proceedings of EVIA 2008.
8–9.
[17] Tetsuya Sakai, Jin Young Kim, and Inho Kang. 2023. A Versatile Framework for
Evaluating Ranked Lists in terms of Group Fairness and Relevance. ACM TOIS
(2023). https://doi.org/10.1145/3589763[18] Tetsuya Sakai, Sijie Tao, Nuo Chen, Yujing Li, Maria Maistro, Zhumin Chu, andNicola Ferro. 2023. On the Ordering of Pooled Web Pages, Gold Assessments,
and Bronze Assessments. ACM TOIS (2023). https://dl.acm.org/doi/pdf/10.1145/
3600227[19] Tetsuya Sakai, Sijie Tao, Zhumin Chu, Maria Maistro, Yujing Li, Nuo Chen, Nicola
Ferro, Junjie Wang, Ian Soboroff, and Yiqun Liu. [n.d.]. Overview of the NTCIR-16
We Want Web with CENTRE (WWW-4) Task. In Proceedings of NTCIR-16. 234–
235. https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings16/pdf/ntcir/01NTCIR16-OV-WWW-SakaiT.pdf[20] Tetsuya Sakai and Zhaohao Zeng. 2020. Retrieval Evaluation Measures that
Agree with Users’ SERP Preferences: Traditional, Preference-based, and Diversity
Measures. ACM TOIS 39, 2 (2020), Article 14.
[21] Yiteng Tu, Haitao Li, Zhumin Chu, Qingyao Ai, and Yiqun Liu. 2023. THUIR at
the NTCIR-17 FairWeb-1 Task: An Initial Exploration of the Relationship Between
Relevance and Fairness. In Proceedings of NTCIR-17. https://doi.org/10.20736/
0002001317

This paper reported on the official results of the NTCIR-17 FairWeb1 task. The most remarkable results are as follows.
• In terms of relevance measures, the THUIR runs (e.g. THUIRQD-RR-4 - a “PM2 and dense relevance” run [21]) perform
well on the entire topic set, and on each of the three topic
subsets.
• In terms of group fairness (GF) measures, the THUIR runs
perform well as well, on all three topic subsets (M, R, Y), withall attribute sets.
• Hence, in terms of GFR as well, the THUIR runs are thewinners (in terms of average performance) on all three topic
subsets.
We also demonstrated how the GFR framework [17] works using a
THUIR run and a baseline run.
We plan to propose the FairWeb-2 task for NTCIR-18 and tocontinue studying the group-fair web search problem, in particular
to see whether we can reproduce the best THUIR runs on new dataand even improve upon them. Furthermore, we plan to introduce agroup-fair conversational search subtask, which requires systems
to return textual turns instead of ranked lists of web pages. Moredetails can in be found in Sakai [15].

DISCLAIMER
Certain companies and products are identified in this paper in orderto specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by
NIST, nor is it intended to imply that the products or companies
identified are necessarily the best available for the purpose.

REFERENCES[1] Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, andEmine Yilmaz. 2008. Relevance Assessment: Are Judges Exchangeable and DoesIt Matter?. In Proceedings of ACM SIGIR 2008. 667–674.
[2] Olivier Chapelle, Donald Metzler, Ya Zhang, and Pierre Grinspan. 2009. Expected
Reciprocal Rank for Graded Relevance. In Proceedings of ACM CIKM 2009. 621–
630.
[3] Fumian Chen and Hui Fang. 2023. UDInfoLab at the NTCIR-17 FairWeb-1 Task.
In Proceedings of NTCIR-17. https://doi.org/10.20736/0002001300[4] Sachin Pathiyan Cherumanal, Kaixin Ji, Danula Hettiachchi, Johanne R. Trippas,
Falk Scholer, and Damiano Spina. 2023. RMIT_IR at the NTCIR-17 FairWeb-1
Task. In Proceedings of NTCIR-17. https://doi.org/10.20736/0002001315[5] Zhumin Chu, Tetsuya Sakai, Qingyao Ai, and Yiqun Liu. 2023. Chuweb21D:
A Deduped English Document Collection for Web Search Tasks. In In Annual
International ACM SIGIR Conference on Research and Development in Information
Retrieval in the Asia Pacific Region. ACM.
[6] Fan Li, Kaize Shi, Nuo Chen Kenta Inaba, Sijie Tao, and Tetsuya Sakai. 2023.
RSLFW at the NTCIR-17 FairWeb-1 Task. In Proceedings of NTCIR-17. https://doi.org/10.20736/0002001303[7] Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane
Culpepper, and Alistair Moffat. 2020. CC-News-En: A large English news corpus. In Proceedings of the 29th ACM International Conference on Information &Knowledge Management. 3077–3084.
[8] Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. 2007. Detecting nearduplicates for web crawling. In Proceedings of the 16th international conference on
World Wide Web. 141–150.
[9] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang,
Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring theLimits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv
e-prints (2019). arXiv:1910.10683[10] Tetsuya Sakai. 2014. Metrics, Statistics, Tests. In PROMISE Winter School 2013:
Bridging between Information Retrieval and Databases (LNCS 8173). 116–163.
[11] Tetsuya Sakai. 2018. Comparing Two Binned Probability Distributions for Information Access Evaluation. In Proceedings of ACM SIGIR 2018. 1073–1076.

305