NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan
DOI: https://doi.org/10.20736/0002001303

RSLFW at the NTCIR-17 FairWeb-1 Task
Fan Li

Kaize Shi

Kenta Inaba

Waseda University
Japan, Tokyo
lif@akane.waseda.jp

Waseda University
Japan, Tokyo
edmund_kz@akane.waseda.jp

Waseda University
Japan, Tokyo
kinaba@ruri.waseda.jp

Sijie Tao

Nuo Chen

Tetsuya Sakai

Waseda University
Japan, Tokyo
tsjmailbox@ruri.waseda.jp

Waseda University
Japan, Tokyo
pleviumtan@toki.waseda.jp

Waseda University
Japan, Tokyo
tetsuyasakai@acm.org

ABSTRACT

the approaches is introducing fairness into ranked list after ranked
list is built which is called reranking. This approach will make sure
to appear protected group in the ranked list. Zehlike et al.[5] suggested reranking method. This research presents a solution for the”Fair Top-k Ranking problem,” focusing on selecting a subset of kcandidates from a pool of n > k, optimizing utility while adhering
to group fairness. The study introduces a fresh definition of ranked
group fairness, maintaining protected candidates’ proportional representation in the top-k ranking’s prefixes. It employs two utility
criteria: top-k candidates outperform non-included candidates, andhigher-ranked candidates in the top-k surpass their counterparts.
The resulting algorithm efficiently generates Fair Top-k Rankings,
effectively minimizing biases while maximizing utility, marking anovel advancement in addressing biases in ranked lists.

The RSLFW team participated in the NTCIR-17 FairWeb 1 Task.
This paper reports our approach to solving the problem and discusses the official results. We applied several different methods togenerate 5 runs, including PM-1, PM-2 and DetGreedy algorithm,
all of which are post-processing approaches. We also utilized COIL
(Contextualize Inverted List) as the RSLFW baseline. By combining
official baseline and COIL baseline with different fairness-related
algorithm, we analyzed the results of those methods. Our reranked
run outperforms the baseline, resulting in an improved GFR score.

KEYWORDS
query expansion, learning to rank, reranking

TEAM NAME

3

RSLFW

This section describes out approach to solving the problem. In theprovided dataset, documents lack explicit group labels, complicating
the use of group-based approaches. To address this challenge:
For the R topic, we employ Spacy1 to perform Named Entity
Recognition (NER), subsequently assigning labels: female, male,or unknown. This assignment is based on the proportionality of
named entities corresponding to each gender. For the M topic, Spacyis once again utilized, this time to extract regional information. The
Y topic remains unaltered, with no additional processing.

1

INTRODUCTION

The RSLFW team participated in the NTCIR-17 FairWeb 1 Task[4].
This paper reports our approach to solving the problem and discusses the official results.
Our methods can be primarily divided into two directions. The
first direction is based on pretrained transformer model called COIL
with manual query operation. COIL is one modified BERT-likemodel that utilizes storing representation vectors into inverted
lists. We take the COIL retrieved result as the baseline for RSLFW
and employ an election-based approach for fairness-related postprocessing. The second direction involves applying Score maximizing greedy mitigation algorithm on the basis of official baseline.
We ultimately observe the differences in results between these two
directions.
The rest of the paper is divided into four parts as follows. Section
2 introduces related work, and section 3 describes our methods indetail. Results and the reasons behind them are reported in section
4. Finally, section 5 concludes our work.

2

3.1

METHODS

Manual Operation of the query

Given that the search results for the M topic should encompass
movie-related entities, and those for the Y topic should feature
a YouTube video, we made manual adjustments to enhance theaccuracy of our search queries. Specifically, for the M topic, we appended the keyword “IMDb” to the end of each query. Similarly, for
the Y topic, “YouTube” was concatenated at the query’s conclusion.
The queries pertaining to the R topic were retained as originallyprovided.

3.2

RELATED WORK

COIL Baseline

Recently, information retrieval systems have transitioned from exact lexical matching techniques like BM25 to pretrained transformer
models such as BERT for soft semantic matching. However, while
the former lacks context sensitivity, the latter compromises computational efficiency. Gao et al.[2] introduced COIL, a novel method

Fairness in the field of information retrieval (IR) has been gathering
a lot of attentions these days. This concept does not only enables
people to reach out relevant documents or pages but also find what
would have been dismissed. Learning to rank (LtR) is one of theframework to improve ranked list. Some approaches among LtR
have been discussed to introduce fairness into ranked list. One of

1 https://spacy.io/

310

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japanthat combines the strengths of both techniques. By encoding document tokens to compute contextualized vector representations with
BERT, these vectors are stored in an inverted index corresponding
to the token, accompanied by the document id. During a search
query, COIL calculates scores using these vector representations,and leverages the special [CLS] token to address vocabulary mismatches. While COIL has outpaced traditional and modern retrieval
methods and minimized latency, it struggles with documents exceeding BERT’s 512-token limit. To counter this, documents aresegmented into 510-token chunks for encoding and then reassembled, ensuring comprehensive representation in the inverted index.

3.3

by accommodating the likelihood that a document might pertain to
multiple topics simultaneously. PM-2 operates on the assumptionthat all documents 𝑑 𝑗 in 𝐷 are relevant to every aspect 𝑡𝑖 in 𝑇 , eachhaving a relevance probability 𝑃 (𝑑 𝑗 |𝑡𝑖 ). When identifying the next
optimal document, the factor 𝜆 balances relevance to the prime
aspect 𝑡𝑖∗ against broader aspect relevance. Unlike PM-1, where𝑠𝑖 denotes the count of seats occupied by 𝑡𝑖 , in PM-2, 𝑠𝑖 is better
perceived as the proportion of the seat that 𝑡𝑖 occupies, since a
selected document 𝑑 ∗ is presumed relevant to all aspects and hence
multiple aspects share the seat.
We believe that both these approaches are suitable for our fairranking problem, where we expect different groups can secure theirsupposed-to-be “seats” in a document ranking. The details of thealgorithm are elaborated in Algorithm 1.

An Election-based approach

Algorithm 1 PM Algorithm
1: 𝑠𝑖 ← 0, ∀𝑖

3.4

2: for all seats in the ranked list 𝑆 dofor all aspects 𝑡𝑖 ∈ 𝑇 do
quotient[i] = 2𝑠𝑣𝑖 𝑖+1
5:
end for6:𝑖 ∗ ← arg max𝑖 quotient[𝑖]7:if mode = 1 then8:𝑑 ∗ ← pop 𝑀𝑖 ∗9:𝑠𝑖 ∗ ← 𝑠𝑖 ∗ + 1
10:else if mode = 2 then11:𝑑 ∗ ← arg max𝑑 𝑗 ∈𝑅 𝜆 × quotient[𝑖 ∗ ] × 𝑃 (𝑑 𝑗 |𝑡𝑖 ∗ ) + (1 −Í∗𝜆) 𝑖≠𝑖 quotient[𝑖] × 𝑃 (𝑑 𝑗 |𝑡 𝑗 )12:
𝑆 ← 𝑆 ∪ {𝑑 ∗ }13:
𝑅 ← 𝑅 \ {𝑑 ∗ }14:
for all aspects 𝑡𝑖 ∈ 𝑇 do
𝑃 (𝑑 ∗ |𝑡 )15:
𝑠𝑖 ← 𝑠𝑖 + Í 𝑃 (𝑑 ∗𝑖|𝑡 )
3:

4:

𝑡𝑗

DetGreedy-based approach

DetGreedy is proposed by LinkedIn[3] for mitigating bias in searchand recommendation systems. The core idea of DetGreedy algorithm is to prioritize satisfying minority groups. For the different
attributes in sorting, a range of feasible sorting algorithms is defined
as shown below:∀𝑘 ≤ |𝜏𝑟 | & ∀𝑎𝑖 ∈ 𝐴, count𝑘 (𝑎𝑖 ) ≤ ⌈𝑝 (𝑎𝑖 ) · 𝑘⌉ and,(1)∀𝑘 ≤ |𝜏𝑟 | & ∀𝑎𝑖 ∈ 𝐴, count𝑘 (𝑎𝑖 ) ≥ ⌊𝑝 (𝑎𝑖 ) · 𝑘⌋(2)

where 𝑘 is number of desired results, 𝜏𝑟 represents ranked list of
candidates for search request 𝑟 , and 𝐴 is set of fairness attributes
to consider. 𝑝 (𝑎𝑖 ) is desired distribution of attribute 𝑎𝑖 .
A feasible fairness ranking algorithm ensures that the protected
attributes have lower and upper limits, referred to as minimum representation and maximum representation. Once ranking list starts
to appear that the current number of attributes is less than the minimum representation or greater than the maximum representation,
it is considered to be infeasible.
DetGreedy algorithm performs cumulative counting analysis oneach row of the initial ranking result. When the value of current
attribute violates the minimum representation requirement, thealgorithm selects the attribute with the highest relevance scorefrom those attribute clusters which are less than minimum representation. If current attribute value meets minimum representation,
the algorithm then selects the attribute value with the highest relevance score from the clusters that have not met the maximum
representation. Algorithm 2 shows the detailed information of DetGreedy.
In contrast, DetConstSort algorithm, which is also introduced by
LinkedIn[3], incrementally increases the representation of attributevalues and then inserts candidates based on their scores. Specifically,
DetConstSort traverses through the results from the beginning until
a attribute value meeting the minimum representation is found.
The next candidate with that attribute value is selected. If inserting
this candidate violates the maximum representation, the algorithm
attempts to swap this candidate to a position that satisfies the
maximum representation requirement, while maintaining a higherscore for results in earlier positions.
Through extensive experimentation, it has been demonstrated
that the DetGreedy algorithm performs the best in terms of fairness,𝑗end for17:
end if
18: end for
16:

Dang and Croft[1] introduce an innovative concept in the realmof search results diversification: “diversity by proportionality.” Contrary to traditional diversification techniques, they assert that a
result list achieves optimum diversity when it furnishes documents
related to a query’s associated topics in numbers proportionate toeach topic’s prevalence. The inspiration for their framework stems
from electoral systems where seats are distributed among politicalparties based on the votes they garner.
Drawing from the Sainte-Laguë Method used in electoral contexts, they introduced PM-1. In this method, each spot on the search
result list is assessed iteratively. For each position, PM-1 computes
a quotient for all topics based on their relative popularity, and the
topic with the highest quotient is selected. The top-ranking document for that topic then claims that position in the list. In PM-1,
represented by choose mode=1, 𝑣𝑖 and 𝑠𝑖 denote the expected number of documents for aspect 𝑡𝑖 and the count of documents alreadyassigned to 𝑡𝑖 respectively. The set 𝑀𝑖 is comprised of documents{𝑑 (1), 𝑑 (2), . . . , 𝑑 (𝑙𝑖 )} related to aspect 𝑡𝑖 , with the top document
for 𝑡𝑖 being the document with higher relevance score in 𝑀𝑖 .
Building on this concept, a probabilistic interpretation of theSainte-Laguë Method, known as PM-2, further refines this approach

311

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Algorithm 2 Score Maximizing Greedy Mitigation Algorithm (DetGreedy)

metrics. This suggests that the run without reranking is more relevant. Conversely, RSLFW-Q-MN-3 and RSLFW-Q-MN-2 consistently achieve superior group fairness scores. This pattern aligns
with the commonly understood "relevance and fairness trade-off."
In the context of reranked runs, RSLFW-Q-RR-4 and RSLFW-QRR-5, both based on the official run, exhibit distinct performances.
Despite the fact that both of them show no statistically significant
different from each baseline, RSLFW-Q-RR-5 demonstrates a superior relevance score. On the other hand, RSLFW-Q-RR-4 excels
in the group fairness score, earning the highest GFR score among
all submitted runs as shown in Table3. Comparing RSLFW-Q-RR-5
and its origin, baseline, our reranking method slightly improves
most of all the scores. The reranking method also improves therelevance scores as shown in Table2.
For the R topic, as depicted in Tables4, the results bear a resemblance to those of the M topic. The exception is RSLFW-Q-RR-5,
which emerges as the top performer in the GFR score among all
the submitted runs.
However, the Y topic results, presented in Rable5, deviate slightly.
Here, RSLFW-Q-MN-3 and RSLFW-Q-MN-2 outperform RSLFWQ-MN-1 in both relevance and group fairness scores. Moreover,
RSLFW-Q-RR-4 secures the leading position for the GFR scoreamong all the submitted runs.

1: foreach 𝑎𝑖 ∈ 𝑎, counts[𝑎𝑖 ] B 0
2: rankedAttList B []; rankedScoreList B []3: for 𝑘 ∈ {1, . . . , 𝑘 max } dobelowMin := {𝑎𝑖 : counts[𝑎𝑖 ] < ⌊𝑘 · 𝑝𝑎 ⌋}belowMax := {𝑎𝑖 : counts[𝑎𝑖 ] ≥ ⌊𝑘 ·𝑝𝑎𝑖 ⌋ and counts[𝑎𝑖 ] <⌈𝑘 · 𝑝𝑎𝑖 ⌉}6:
if belowMin ≠ ∅ then7:
nextAtt := arg max𝑎𝑖 ∈belowMin 𝑠𝑎𝑖 ,counts[𝑎𝑖 ]
8:
else
9:
nextAtt := arg max𝑎𝑖 ∈belowMax 𝑠𝑎𝑖 ,counts[𝑎𝑖 ]4:5:rankedAttList[k] B nextAttrankedScoreList[k] B 𝑠 nextAtt,counts[nextAtt]
12:
counts[nextAtt]++
13: return [rankedAttList,rankedScoreList]
10:
11:

provided strict minimum representation requirements are not imposed for every attribute. Consequently, we choose the DetGreedy
algorithm as our baseline-based reranking method.

4 EXPERIMENTS
4.1 Run Description

Table 2: Evaluation results over the 45 topics for our submitted runs. The baseline is run.bm25-depThre3-Q

The RSLFW team contributed five runs for each topic category. Of
these, two were designated as ’RR’ runs, signifying the application of a reranking algorithm on the official run. The other threewere reranked based on the COIL baseline. For the COIL baseline
construction, we took inspiration from Gao et al.’s work and utilized BERT-base (uncased version with 768 CLS dimensions and110M parameters) as our pretrained language model. This model
was trained using the MSMARCO passage dataset, a compilationof user queries sourced from Bing’s search logs combined withpassages from web documents. To ensure computational efficiency
with COIL, we first curated a sub-corpus, comprising the top 1,000
most relevant documents for each topic, using the BM25 algorithm
implemented in Anserini. The details of each run are outlined inTable 1.

Run
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5
baseline

4.2

Baseline
COIL
COIL
COIL
Official
Official

iRBU
0.35100.32980.22710.47370.49730.4242

Table 3: Evaluation results over the 15 M topics for our submitted runs. The baseline is run.bm25-depThre3-Q
Run
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5
baseline

Table 1: Description of RSLFW Team’s Runs
Run IDRSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5

ERR
0.12450.10670.07700.14850.18470.1390

Reranking Algorithm
None
PM1
PM2
PM2
DetGreedy Algorithm

ERR
0.18930.14890.14890.16200.20440.1712iRBU
0.42680.42500.42500.54630.56740.5035

GFR
0.35810.3756
0.37560.49960.49490.44844.2.2 Topic Analysis. We conducted a per-topic analysis for further
discussion. The analysis is based on GFR, and the Bronze-All file
was used as the relevance assessment. To prevent redundancy, fromthis point forward, RSLFW-Q-RR-5 is abbreviated as Run 5 andrun.bm25-depThre3-query as baseline, and so on. Table 6 shows the
topics and GFR scores where RSLFW-Q-RR-5 significantly outperformed the baseline. The topic with the largest difference between
two runs was "car racing movies." Only five topics’ scores decreased
and the mean difference is -0.014. The reranking method does notappear to have a negative impact.

Results

4.2.1 Evaluation results. Based on the results for the M topic presented in Table3, a comparison between RSLFW-Q-MN-1 and its
counterparts, RSLFW-Q-MN-3 and RSLFW-Q-MN-2, reveals that
RSLFW-Q-MN-1 consistently scores higher in both ERR and iRBU312

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 4: Evaluation results over the 15 R topics for our submitted runs. The baseline is run.bm25-depThre3-Q
Run
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5
baseline

ERR
0.10210.08900.00000.14780.2131
0.1989iRBU
0.37000.30840.00000.4974
0.54880.54895

GFR
0.36220.30270.00000.48070.50100.5064

REFERENCES[1] Van Dang and W. Bruce Croft. 2012. Diversity by proportionality: an electionbased approach to search result diversification. In Annual International ACM
SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:11318288
COIL: Revisit Exact[2] Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021.
Lexical Match in Information Retrieval with Contextualized Inverted List.
arXiv:2104.07186 [cs.IR][3] Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. 2019. FairnessAware Ranking in Search & Recommendation Systems with Application to
LinkedIn Talent Search. In Proceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD’19). Association for Computing Machinery, New York, NY, USA, 2221–2231.
https://doi.org/10.1145/3292500.3330691[4] Sijie Tao, Nuo Chen, Tetsuya Sakai, Zhumin Chu, Nicola Ferro, Maria Maistro, Ian
Soboroff, and Hiromi Arai. 2023. Overview of the NTCIR-17 FairWeb-1 Task. In
NTCIR-16 to appear.
[5] Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed,and Ricardo Baeza-Yates. 2017. FA*IR: A Fair Top-k Ranking Algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management(Singapore, Singapore) (CIKM ’17). Association for Computing Machinery, New
York, NY, USA, 1569–1578. https://doi.org/10.1145/3132847.3132938

Table 5: Evaluation results over the 15 Y topics for our submitted runs. The baseline is run.bm25-depThre3-Q
Run
RSLFW-Q-MN-1
RSLFW-Q-MN-2
RSLFW-Q-MN-3
RSLFW-Q-RR-4
RSLFW-Q-RR-5
baseline

ERR
0.08220.08220.08220.13570.13650.0471iRBU
0.25620.25620.25620.37750.37570.2202

GFR
0.23810.23810.2381
0.34280.34080.2121

Table 6: GFR of the topics where Run 5 significantly outperformed the baseline
topic
M007
Y001
Y002

query
car racing movies
Bacharach/David covers
Beatles covers

Run 5
0.45810.44390.4001baseline
0.00000.00000.0432

CONCLUSIONSThe RSLFW team participated in the NTCIR-17 Fair Web(FairWeb1) task. We submitted five runs. Two of them are based on COIL
and the others are reranked on the official baseline. We discussed
the trade-offs between relevance and fairness. When comparing
our reranked run with the baseline, we observed an improvementin the GFR score.

difference
0.45810.44390.3569313