NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
We Want Web with CENTRE (WWW-4) Task
Tetsuya Sakai, Sijie Tao

Zhumin Chu

Maria Maistro

Waseda University, Japantetsuyasakai@acm.org
tsjmailbox@ruri.waseda.jp

Tsinghua University, P.R.C.
chuzm19@mails.tsinghua.edu.cn

University of Copenhagen, Denmark
mm@di.ku.dk

Yujing Li, Nuo Chen

Nicola Ferro

Junjie Wang

Waseda University, Japanliyujing@ruri.waseda.jp
pleviumtan@toki.waseda.jp

University of Padua, Italy
ferro@dei.unipd.it

Waseda University, Japan
wjj1020181822@toki.waseda.jp

Ian Soboroff

Yiqun Liu

NIST, USAian.soboroff@nist.gov

Tsinghua University, P.R.C.
yiqunliu@tsinghua.edu.cn

ABSTRACT

Table 1: WWW-4 timeline (time zone: UTC+9).

This is an overview of the NTCIR-16 We Want Web with CENTRE(WWW-4) task, the fourth round of an evaluation series that aims toquantify the progress and reproducibility of web search algorithms
in offline ad hoc retrieval settings. For WWW-4, we introduced a
new English web corpus, which we named Chuweb21. Moreover,in addition to bronze relevance assessments (i.e., those given byassessors who are neither topic creators nor topic experts), wecollected gold relevance assessments (i.e., those given by topic
creators). We received 18 runs from 4 teams, including two runs
from the organiser team. We describe the task, data, evaluation
measures, and report on the official evaluation results.

1

November 1, 2021
November 4, 2021
December 15, 2021
December 21-January 20, 2021
February 1, 2021

Topics released
BM25 Baseline run released
Run submissions due
Relevance assessments
Evaluation results released

Table 2: WWW-3 run statistics. Besides these 16 runs, wehave two organisers runs: ORG-TOPICDEV and baseline.
Team
KASYS
SLWWW
THUIR
total

INTRODUCTION

This paper presents an overview of the NTCIR-16 We Want Web
with CENTRE (WWW-4) Task,1 the fourth round of an evaluation
series that aims to quantify the progress and reproducibility of web
search algorithms in offline ad hoc retrieval settings.
The We Want Web task (WWW-1) was launched at NTCIR-13 in2017 [9], in response to the termination of the web track at TREC
2014.2 WWW-1 received a total of 32 runs from five teams for the
Chinese and English subtasks.
The NTCIR-14 WWW-2 task was held in 2019 [10], and received 31 runs from five teams for the same subtasks. Also, NTCIR14 hosted the first CENTRE (CLEF/NTCIR/TREC Reproducibility)
task [15], which was a “metatask” that spanned CLEF, TREC, and
NTCIR [5–7, 21].
As NTCIR-14 CENTRE attracted only one participating team, for
NTCIR-15 the WWW and CENTRE joined forces to organise the
We Want Web with CENTRE (WWW-3) task. WWW-3 received
a total of 48 runs from nine teams for the Chinese and English
subtasks [17].
As NTCIR-15 WWW-3 attracted only two participating teams for
the Chinese subtask, we focused on English web search at WWW-4.

NEW
5
4
5
14

REV
1
N/A
N/A
1

REP
N/A
1
0
1

total
6
5
5
16

Moreover, instead of using Clueweb12-B133 again, we constructed a
new English web corpus for the task, which we call Chuweb21. Another new feature of WWW-4 is that, in addition to bronze relevanceassessments (i.e., those given by assessors who are neither topic
creators nor topic experts), we collected gold relevance assessments(i.e., those given by topic creators) [1].
Table 1 shows the timeline of the WWW-4 task. Table 2 shows
names of the participating teams and the number of runs submitted
to the task. It is unfortunate that there was no participation besides
University of Tsukuba (KASYS) whose runs served as the targetof reproducibility experiments and two teams from the organisers’
affiliations: SLWWW (Waseda University) and THUIR (Tsinghua
University). As the table shows, we had three run types: NEW, REV,
and REP, as we shall discuss in Section 2.
The remainder of this paper is organised as follows. Section 2
describes the task, and Section 3 describes the WWW-4 data. Section 4 describes the evaluation measures we use for quantifying
retrieval effectiveness and reproducibility. Sections 5 and 6 report

1 http://sakailab.com/www4/
2 https://twitter.com/djoerd/status/536128465276530688

3 http://lemurproject.org/clueweb12/

231

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japanon the retrieval effectiveness and reproducibility results, respectively. Finally, Section 7 concludes this paper.

2(5.2%), .de (4.2%), .net (3.7%), .uk (2.3%), .jp (1.9%), .fr (1.8%), .it (1.8%)
and .nl (1.6%). To avoid the inclusion of many non-English contents
in the corpus, we only retained web pages under the .com, .org and.net domains. Specifically, we adopted the following constraints tofilter for useful html pages:

TASK

The WWW-4 task is an ad hoc English web search task. Three typesof runs were allowed.

• The root domain must be one of the .com, .org, .net domains;• The WARC-Type must be “response” (actually the web pages
as we need);• The character length of HTML content must be larger than
1,000;• The probability that the document content belongs to English
is greater than 0.99.

REV (revived) run This is a run kindly provided by KASYS(University of Tsukuba). At the NTCIR-15 WWW-3 English
subtask [17], their run KASYS-E-CO-NEW-1 [20] was the top
performer and therefore we treat this run as the SOTA (stateof-the-art) from WWW-3. It uses a BERT-based approach
proposed in Yilmaz et al. [27]. We asked KASYS to use theexact algorithm used at WWW-3 to process the new WWW4 topics to from a revived run. The resultant WWW-4 run iscalled KASYS-CO-REV-6 [23].
NEW runs These runs are the regular adhoc runs designed toadvance the SOTA. If a NEW run substantially outperforms
the above REV run on the WWW-4 test collection, that suggests that we have a new SOTA. In this way, we can examine
if we are seeing real technological progress.
REP runs These runs aim at reproducing what KASYS did at
WWW-3 to generate the WWW-3 run KASYS-E-CO-NEW-1.
Since the same algorithm was used to generate the WWW-4
run KASYS-CO-REV-6, we can discuss the reproducibilityof the KASYS method by simply comparing the REP runs
with KASYS-CO-REV-6 on the new WWW-4 test collection.
Unfortunately, however, only one REP run was submitted to
the WWW-4 task [22].

We used 4 servers (32 processes) running for about two weeks tocomplete the aforementioned data cleaning jobs. The final Chuweb21
corpus contains 82, 451, 337 (9.6% of the subdata) HTMLs or 1.69
TiB of compressed content. Similar to the ClueWeb12-B13 data,
Chuweb21 has been reorganized with the “warc.tar” format to facilitate the users. The dataset is already accessible to the public for
academic usage.6

3.2

Topics

3.2.1 Topic Set Size. As with the previous WWW rounds, we decided on the number of topics to create using Sakai’s topic set
size design tool for comparing 𝑚 = 2 systems with a 𝑡-test (orANOVA) [13].7
From the 160×36 topic-by-run score matrices from the WWW-3
English subtask, we obtained the variance estimates of the four evaluation measures [17] as residual variances from two-way ANOVA(𝑉𝐸2 ) [13, p. 120]: nERR (normalised Expected Reciprocal Rank) [12]had the largest variance (0.0284) and iRBU (intentwise Rank-Biased
Utility) [18] had the smallest variance (0.00716). Under Cohen’s fiveeighty convention (𝛼 = 0.05, 𝛽 = 0.20),8 the topic set size design
tool tells us the following:

Each team was allowed to submit up to five NEW/REP runs. KASYS
submitted their REV runs as their sixth run.
Compared to the previous WWW tasks, WWW-4 is different in
two ways. First, we use a new target English web corpus, Chuweb21,
which we constructed as described in Section 3.1. Second, this time
we hired gold assessors [1], that is, we collected the relevance assessments from the topic creators. In fact, the first seven authors of thispaper served as the gold assessors! To maintain consistency withthe previous WWW tasks, we also hired bronze assessors (i.e., those
who are neither topic creators nor topic experts) [1] to construct
an alternative version of qrels. The details are given in Sections 3.

• We need 44 topics for a minimum detectable difference [13](for 80% statistical power) of 0.1 in terms of nERR;• We need 45 topics for a minimum detectable difference (for80% statistical power) of 0.05 in terms of iRBU.
Based on the above estimates, we decided to construct 50 topics
for the WWW-4 task.

3 DATA
3.1 The Chuweb21 Corpus

3.2.2 Topic Creation. There was an indication that the WWW-3
bronze English relevance assessments contained some noise [11].
Moreover, although we created multiple versions of qrels files at
WWW-3 using two different document ordering strategies for the
relevance assessors (RND and PRI [16]), it was not possible to saywhich versions were correct as all of the assessors involved werebronze assessors. We therefore decided to construct a gold qrels
file along with a bronze one. Gold-relevant documents are what
the topic creators want, and therefore can be treated as the rightanswers, although they are not immune to human errors.
In light of the above situation, the first seven authors of this paper
volunteered to serve as the topic creators and the gold relevanceThis corpus was named after one of the task organisers, Zhumin
Chu4 . Chuweb21 was generated based on the April 2021 blockof Common Crawl dataset5 . As the complete data block is toolarge to conduct the downstream data cleaning and indexing work,
we sampled the web pages crawled between 2021-04-10 10:58:31
and 2021-04-11 11:56:10. This part of the data (denoted as subdata
in the following content) contains 3, 402, 457 different domains
and 858, 616, 203 different web pages, which occupies the space of
5.66TiB.
We grouped the subdata by root domain and found that the
top-10 frequent root domains include .com (43.7%), .org (5.7%), .ru

6 https://drive.google.com/drive/folders/11hi_R6cSIHEZx3QwyG5KQjgRVmxXhWta?usp=sharing
7 http://www.f.waseda.jp/tetsuya/samplesizeANOVA2.xlsx

4 This is not a misspelling of Clueweb.

8 This means we want the Type I and Type II error probabilities to be 5% and 20%,5 https://commoncrawl.org/2021/04/april-2021-crawl-archive-now-available/

respectively.

232

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan<queries><query><qid>0201</qid><content>Timnit Gebru Google</content><description>I want to know the details regarding Google's firing of Dr. Timnit Gebru.</description></query><query><qid>0202</qid><content>New Orleans restaurants</content><description>Tell me about good restaurants in New Orleans.</description></query>
Figure 1: The top part of the WWW-4 topic file.
assessors. Each of the seven organisers tried to create realistic topics,
i.e., those based on their actual information needs. The first authorof this paper was responsible for creating eight topics and providing
gold assessments for them; similarly, the other six authors eachhandled seven topics.
To develop the topics, the seven organisers used a browserbased topic development tool to conduct some pilot searches on
the Chuweb21 corpus to ensure that there is at least one relevant
document. They were allowed to formulate and reformulate theirown queries.
The WWW-4 test topic file is publicly available.9 Figure 1 shows
the top part of this file.

From the 18 runs (16 participant runs plus the 2 organiser runs),
we formed a depth-60 pool for each topic, and obtained a totalof 10,333 topic-document pairs to judge (206.7 docs per topic onaverage). We created the following two types of pool file for each
topic:
RND The pooled documents are randomly ordered;
PRI The pooled documents are ordered by pseudorelevance,
based on the number of runs that returned that document
and the ranks of that document in those runs, using the
NTCIRPOOL script [16].10
Elsewhere, we plan to report on a study that compares RND-based
and PRI-based relevance assessments from gold assessors, to follow
up on the work of Sakai, Tao, and Zeng [16] that compared RND
and PRI under the bronze setting.

3.2.3 Topic Set File. The content field represents the Assessor
query that the topic creator is likely to enter, and the description
field concisely describes the topic creator’s information need. If a
run file name contains “CO,” that means only the content field wasused as the input to the system; if it contains “CD,” that means bothcontent and description fields were utilised.

3.3

3.5

Organisers’ Runs

During topic development, the topic creators (i.e., seven organisers)
identified at least one relevant document for each topic. We createda “manual” run from these documents, which we call ORG-TOPICDEV.
This run contains only 97 topic-document pairs for the 50 test topics,
and the documents for each topic are not necessarily sorted byperceived relevance.
We also provided an Anserini-based [25] vanilla BM25 baseline
run together with the contents of the retrieved documents to participants, so that the participants can optionally rerank the baseline
to produce their own runs. This run is a CO (content-only) run, andis simply called baseline.

3.4

Relevance Assessments

For evaluating the WWW-4 runs, we constructed 2 versions of qrels(relevance assessment) files: the Gold version and the Bronze-All
version.
The Gold file is based on the relevance assessments given by
the topic creators, i.e., the first seven authors of this paper. By
definition, each topic was judged by exactly one assessor. For each
topic, each gold assessor processed either the RND pool or the PRI
pool assigned at random.
The Bronze-All file is actually the result of merging two different
versions of relevance assessments given by bronze assessors (i.e.,
those who are neither topic creators nor topic experts [1]). At
Waseda University, Japan, five English-course computer science
students were hired as bronze assessors, as in previous English
WWW subtasks. At Tsinghua University, China, five more bronze
assessors were hired through a Chinese company. All 10 bronze
assessors used the PRI pool files, not the RND pool files; every topic
was assessed by one Waseda assessor and one Tsinghua assessor;
the topics were assigned at random so that each bronze assessor
handled exactly 10 topics.
All gold and bronze assessors used the PLY interface [14, Figure4] for the relevance assessments, and each document was labeledeither highly relevant, relevant, nonrelevant, or error [17]. These

Runs and Pool Files

Table 3 shows the run names and the system descriptions of the
16 runs submitted by KASYS [23], SLWWW [22], and THUIR [26].
Note that SLWWW-CO-REP-1 is the only REP run. That is, this is the
only run that tackled the reproducibility problem.
9 https://waseda.box.com/www4topicsxml

10 https://research.nii.ac.jp/ntcir/tools/ntcirpool-en.html

233

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 3: 16 participant runs and their system description (SYSDESC) fields.
KASYS-CD-NEW-1

KASYS-CD-NEW-3
KASYS-CD-NEW-5

KASYS-CO-NEW-2

KASYS-CO-NEW-4

KASYS-CO-REV-6
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-4
SLWWW-CO-NEW-5
SLWWW-CO-REP-1
THUIR-CO-NEW-1

THUIR-CO-NEW-2

THUIR-CO-NEW-3

THUIR-CO-NEW-4

THUIR-CO-NEW-5

On the topic that include proper noun, We rerank top 10 from documents retrieved from bm25. Other then that,
We use the ranking from bm25 directly. For the re-ranking method, we use BERT fine-tuned on the SQuAD 2.0
dataset. For question generation, we translate www4 topics to question manually.
We rerank top 10 from documents retrieved from bm25. For the re-ranking method, we use BERT fine-tuned on
the SQuAD 2.0 dataset. For question generation, we translate www4 topics to question manually.
We rerank top 100 from documents retrieved from bm25 on the topic that include proper noun. For the
re-ranking method, we use BERT fine-tuned on the SQuAD 2.0 dataset. For question generation, we translate
www4 topics to question manually.
On the topic that include proper noun, We rerank top 10 from documents retrieved from bm25. For the
re-ranking method, we use BERT fine-tuned on the SQuAD 2.0 dataset. For the question generation, we use
Encoder-Decoder neural machine translation model.
We rerank top 10 from documents retrieved from bm25. For the re-ranking method, we use BERT fine-tunedon the SQuAD 2.0 dataset. For the question generation, we use Encoder-Decoder neural machine translation
model.
Revival of KASYS-E-CO-NEW-1 at NTCIR-15 WWW-3
COIL, contextualized exact lexical match. Split into chunks. Smaller corpus
COIL, contextualized exact lexical match. Split into chunks. Bigger corpus
COIL, contextualized exact lexical match. No splitting. Smaller corpus
Reproduction of PARADE full transformer based model
Rep run of the KASYS system
We first use BM25 to retrieve the top-100 documents of each query, and then use PROP to rerank the
top-100 documents. In the training, we use 206 queries in www1-3 dataset (280 queries totally) as train set
to fine-tune PROP, and the remaining 52 queries as validation set.
We first use BM25 to retrieve the top-100 documents of each query, and then use PROP to rerank the
top-100 documents. In the training, we use all 280 queries in www1-3 dataset as the train set tofine-tune PROP and no validation set.
We first use BM25 to retrieve the top-100 documents of each query, and then use BERT-Prompt to rerank
the top-100 documents. BERT-Prompt is trained by cloze prompt method based on BERT on 206 queries in
www1-3 dataset (280 queries totally) as train set, and the remaining 52 queries as validation set.
The LambdaMART model implemented by Ranklib. We adopt MQ2007&2008, www1-3 as our training
and development datasets. The features contain TF, IDF, TF*IDF, DL, BM25, LM.ABS, LM.DIR, LM.JM
in the fields of content, title, url, anchor text.
The Coordinate Ascent model implemented by Ranklib. We adopt MQ2007&2008, www1-3 as our training
and development datasets. The features contain TF, IDF, TF*IDF, DL, BM25, LM.ABS, LM.DIR, LM.JM
in the fields of content, title, url, anchor text.

labels were mapped to scores of 2, 1, 0, and 0, respectively. Thus theGold qrels file contains 3-point (0,1,2) relevance levels. Similarly,
from Waseda’s and Tsinghua’s bronze assessments, we obtained
a 3-point relevance level file, respectively, which we refer to as
Bronze-Waseda and Bronze-Tsinghua. Finally, for computing theofficial evaluation scores for the WWW-4 task, we created a “BronzeAll” file, by adding the relevance scores from Bronze-Waseda andBronze-Tsinghua and forming 4-point relevance levels.
We report on retrieval effectiveness based on the Gold file andthat based on the Bronze-All file separately: the former represents
results based on “correct” relevance assessments as defined by the
topic creators; the latter is based on views of multiple bronze assessors and are more similar to previous WWW evaluation settings.
Table 4 shows the distribution of relevance labels for each versionof the qrels. Table 5 shows the mean inter-assessor agreement foreach pair of qrels files. Note that each qrels file consists of labelsfrom multiple assessors. Table 6 compares the mean 𝜅’s shown in

Table 5 in terms of statistical significance. These two tables showthat the Gold-Bronze agreements are substantially and statisticallyhighly significantly lower than the Bronze-Bronze (i.e., WasedaTsinghua) agreements.
Tables 7 and 8 examine the per-topic 𝜅’s at the individual assessor level. For example, the “Gold01” row of Table 7 compares
the labels of Gold01 (the first author of this paper) with those of
Waseda and Tsinghua assessors, and shows the mean 𝜅 over theeight topics that he was in charge of. It can be observed that theGold-Tsinghua agreements are higher than the Gold-Waseda agreements for Gold01, Gold 02, Gold03, and Gold07 on average, but not
for Gold04, Gold05, and Gold06. On the other hand, in Table8, theWaseda-Tsinghua (i.e., Bronze-Bronze) agreements are substantiallyhigher than the Bronze-Gold agreements for every Bronze assessor.
In summary, according to the WWW4 data, different versions ofbronze relevance assessments are relatively similar to each other,
but they are substantially different from gold relevance assessments.

234

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 4: Distribution of pooled documents over the relevance levels in the gold and bronze qrels files.
relevance
level
L0
L1
L2
L3
L4
total

Gold
(1 assessor/topic)
7,154
1,806
1,373
N/A
N/A
10,333Bronze-Waseda(1 assessor/topic)
5,584
3,158
1,591
N/A
N/A
10,333

Table 5: Mean per-topic inter-assessor agreement in termsof quadratic weighted Cohen’s 𝜅 (𝑛 = 50 topics).

Bronze-All(2 assessors/topic)
4,900
1,881
1,485
1,241
826
10,333

Table 7: Mean per-topic inter-assessor agreement for eachgold assessor in terms of quadratic weighted Cohen’s 𝜅 (𝑛 = 8
topics for Gold01; 𝑛 = 7 topics for the others). For example,
the labels of Gold01 are compared with those given by the
Waseda and Tsinghua bronze assessors.

qrels version
mean 𝜅
Gold-Waseda
0.242
0.280
Gold-Tsinghua
Waseda-Tsinghua
0.458
Table 6: Comparison of the mean 𝜅’s with a randomised
Tukey HSD test (𝐵 = 5, 000 trials). The effect sizes are basedon the two-way ANOVA residual variance 𝑉𝐸2 = 0.0345 [13].
Gold-Waseda vs. Gold-Tsinghua
Gold-Waseda vs. Waseda-Tsinghua
Gold-Tsinghua vs. Waseda-Tsinghua

Bronze-Tsinghua(1 assessor/topic)
6,571
1,986
1,776
N/A
N/A
10,333sassessor mean 𝜅 (with Waseda) mean 𝜅 (with Tsinghua)
Gold01
0.218
0.306
Gold02
0.258
0.343
0.226
0.283
Gold03
Gold040.326
0.305
Gold050.258
0.221
Gold06
0.154
0.145
0.258
0.350
Gold07
Table 8: Mean per-topic inter-assessor agreement for eachbronze assessor in terms of quadratic weighted Cohen’s 𝜅(𝑛 = 10 topics). For example, the labels of Waseda01 are compared with those given by the Gold and Tsinghua assessors.

𝑝 = 0.679, ES𝐸2 = 0.202
𝑝 ≈ 0, ES𝐸2 = 0.958
𝑝 ≈ 0, ES𝐸2 = 1.160

This suggests that system evaluations based on gold and bronze
relevance assessments may also be substantially different. Section 5
discusses the actual evaluation results based on Gold and Bronze-All
files.
A further investigation showed that the low Gold-Bronze agreements were largely due to the differences in the document presentation order for the assessors: that is, the agreements were extremely
low when the Gold assessors used the RND pool files while theBronze assessors used the PRI pool files. (Recall that all Bronze
assessments are based on PRI bools.) Put another way, the disagreements reflect the differences between RND and PRI document
ordering strategies rather than the differences between Gold andBronze assessors. Details will be reported elsewhere.

assessor
Waseda01
Waseda02
Waseda03
Waseda04
Waseda05
assessor
Tsinghua06
Tsinghua07
Tsinghua08
Tsinghua09
Tsinghua10

4 EVALUATION MEASURES
4.1 Effectiveness Measures

4.2

mean 𝜅 (with Gold)
0.214
0.226
0.247
0.166
0.358
mean 𝜅 (with Gold)
0.352
0.303
0.241
0.201
0.301

mean 𝜅 (with Tsinghua)
0.450
0.459
0.444
0.428
0.507
mean 𝜅 (with Waseda)
0.476
0.485
0.395
0.500
0.432

CENTRE Evaluation Measures

Following the NTCIR-15 WWW-3 task, we use nDCG@10 (MSnDCG@10), In this round of CENTRE, we quantify reproducibility as suggested
Q@10, and nERR@10 [12], and iRBU@10 (with 𝑝 = 0.99) [18]in Breuer et al. [2]12 . We cannot quantify replicability, since thisto evaluate the runs in terms of retrieval effectiveness, using thewould require runs generated by the same system on different test
NTCIREVAL tool with a linear gain value setting.11 According tocollections. Details on reproducibility measures are presented in
the experiments reported by Sakai and Zeng [19], nDCG and iRBU
the following.
outperformed other measures in terms of agreement with users’First, we evaluate whether the reproduced run can retrieve the
SERP preferences.
same exact ranking of documents retrieved by the original run.
12 Note that reproducibility terminology has changed since 2020. In this paper weadopt the updated terminology: https://www.acm.org/publications/policies/artifactreview-badging

11 http://research.nii.ac.jp/ntcir/tools/ntcireval-en.html

235

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

smaller the p-value, the stronger the evidence that 𝑟 and 𝑟 ′ arestatistically significantly different.

We compute Kendall’s 𝜏 union (KTU) [6, 7], which compares
the relative order of documents by computing Kendall’s 𝜏 withrespect to the union of the original and replicated rankings. Thisis necessary since Kendall’s 𝜏 is defined for permutations of items
from the same list [8], while reproduced runs can rank documentsthat were not retrieved by the original run.
Let 𝑟 be the original run and 𝑟 ′ the reproduced run, 𝑟 𝑗 denotes
the ranked list of document ids for topic 𝑗 for the original run andsimilarly 𝑟 𝑗′ is the ranked list of documents for the reproduced run.
KTU is computed as follows:

5

Table 9 shows the official effectiveness results based on the Gold
relevance assessments. Table 10 shows the results of statistical significance tests for Table 9: note that none of the differences in termsof Mean nERR are statistically significant. From the statistical significance point of view, all of the WWW-4 runs except ORG-TOPICDEVare tied even in terms of the other three measures, as none of therun pairs are statistically significantly different except those thatinvolve ORG-TOPICDEV. In other words, from the Gold-based results,
all we know for certain is that ORG-TOPICDEV substantially underperform the others. This is not surprising, since this run contains
only a few documents per topic (See Section 3.3).
The following are probably worth noting from Table 9, however.

(1) consider the union of 𝑟 𝑗 and 𝑟 𝑗′ by removing duplicate entries;(2) consider the rank position of documents from the union in𝑟 𝑗 and 𝑟 𝑗′ ;(3) compute Kendall’s 𝜏 between these two lists of rank positions.
Kendall’s 𝜏 at step 3 is computed as follows:
KTU 𝑗 (𝑟, 𝑟 ′ ) = 𝜏 𝑗 (𝑙, 𝑙 ′ ) = q

𝑃 −𝑄


𝑃 +𝑄 +𝑈 𝑃 +𝑄 +𝑉• None of the runs substantially outperform the REV run,which suggests that we are not seeing any substantial technological advance in this round of the task;• The only REP run (SLWWW-CO-REP-1) performs very similarly to the REV run in terms of all four evaluation measures,
which suggests that the reproducibility effort may be successful to some degree (see Section 6).

(1)

where 𝑙 and 𝑙 ′ are the list of rank positions obtained at step 2,
𝑃 is the total number of concordant pairs, 𝑄 is the total numberof discordant pairs, 𝑈 and 𝑉 are the number of ties, in 𝑙 and 𝑙 ′respectively.
As reported in previous work [2, 6, 7], Kendall’s 𝜏 can be toostrict when comparing 2 lists of documents and is not top heavy.
Therefore, in addition to Kendall’s 𝜏 we also compute Rank-Biased
Overlap (RBO) [24]. RBO for the 𝑗-th topic is computed as follows:
RBO 𝑗 (𝑟, 𝑟 ′ ) = (1 − 𝜙)∞Õ

𝜙 𝑖−1 · 𝑂𝑖

RETRIEVAL EFFECTIVENESS RESULTS

Table 11 shows the official effectiveness results based on the
Bronze-All relevance assessments. Tables 12 and 13 show the results of statistical significance tests for Table 11. It can be observedthat THUIR-CO-NEW-2 is quite successful: this is the only run thatstatistically significantly outperform five other runs in terms ofMean nDCG (Table 12(a)). However, we cannot say with confidence
that this run is now the new SOTA since the difference between
THUIR-CO-NEW-2 and KASYS-CO-REV-6 is not statistically significant. Note also that THUIR-CO-NEW-2 is ranked first in the Gold
Mean Q ranking (Table 9(b)).
Regarding reproducibility, the trend is similar to the Gold-based
results in that SLWWW-CO-REP-1 performs very similarly to KASYSCO-REV-6 in terms of all four evaluation measures. Section 6
discusses reproducibility in more detail.
Table 14 compares the pairs of run rankings in terms of Kendall’s
𝜏. Part (a) compares the Gold-based rankings with different evaluation measures; Part (b) compares the Bronze-All-based rankings
with different evaluation measures; and Part (c) compares the Goldbased and Bronze-All-based rankings for each evaluation measure.
Parts (a) and (b) shows that nDCG and Q are very highly correlated, while the correlation between nERR and iRBU is relatively
low. Part (c) shows that while the Q-based run ranking is the mostrobust when Gold assessments are replaced with Bronze-All ones,
nERR’s run ranking changes completely when this is done: the
95%CI shows that the correlation between the Gold nERR-basedranking and the Bronze-All nERR-based ranking is not statisticallysignificant. Assuming that the Gold-based results are correct, theabove result suggests that (n)ERR-based evaluation with bronze
assessors should be interpreted with caution.
As we have mentioned earlier, it appears that the Gold-Bronze
disagreements are largely due to the use of different document ordering strategies (PRI and RND). We shall report on further findings
elsewhere.

(2)

𝑖=1

where 𝜙 ∈ [0, 1] is a parameter to adjust the measure top-heaviness:the smaller 𝜙, the more top-weighted the measure; and 𝑂𝑖 is the
proportion of overlap up to rank 𝑖, which is defined as the cardinalityof the intersection between 𝑟 𝑗 and 𝑟 𝑗′ up to 𝑖 divided by 𝑖. Therefore,
RBO accounts for the overlap of two rankings and discounts theoverlap while moving towards the end of the ranking, since it ismore likely for two rankings to have a greater overlap when many
rank positions are considered.
In addition to differences in how runs rank documents, we consider the per topic differences in effectiveness scores. Let 𝑀 𝑗 (𝑟 )denote the effectiveness score for topic 𝑗 of the original run. Similarly, let 𝑀 𝑗 (𝑟 ′ ) denote the corresponding score of the reproduced
run. Following CENTRE@CLEF [6, 7], the Root Mean Square Error(RMSE) for replicating absolute per-topic differences is computed
as follows.
v
u
u
t Õ
𝑇
1
𝑅𝑀𝑆𝐸 =(𝑀 𝑗 (𝑟 ′ ) − 𝑀 𝑗 (𝑟 )) 2 ,
(3)
𝑇 𝑗=1
where 𝑇 is the number of topics. Note that 𝑅𝑀𝑆𝐸 focuses on the pertopic measure scores rather than the actual documents retrieved.
Finally, we compare the original and reproduced runs from a statistical point of view [2]. We run a two tailed paired t-test between
𝑀 𝑗 (𝑟 ) and 𝑀 𝑗 (𝑟 ′ ), for 𝑗 ∈ {1, . . . ,𝑇 }. The 𝑝-value returned by thet-test informs on the success of the reproducibility experiment: the236

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 9: Official results based on the Gold file (mean over the 50 WWW-4 test topics).
Run name
SLWWW-CO-REP-1
KASYS-CO-REV-6
THUIR-CO-NEW-2
SLWWW-CO-NEW-4
THUIR-CO-NEW-1
THUIR-CO-NEW-5
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
KASYS-CO-NEW-4
KASYS-CD-NEW-1
KASYS-CD-NEW-3
KASYS-CO-NEW-2
THUIR-CO-NEW-3
baseline
SLWWW-CO-NEW-5
THUIR-CO-NEW-4
KASYS-CD-NEW-5
ORG-TOPICDEVRun name
THUIR-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-2
THUIR-CO-NEW-1
SLWWW-CO-REP-1
KASYS-CO-REV-6
SLWWW-CO-NEW-4
KASYS-CO-NEW-4
THUIR-CO-NEW-5
KASYS-CD-NEW-1
KASYS-CO-NEW-2
KASYS-CD-NEW-3
THUIR-CO-NEW-4
KASYS-CD-NEW-5
baseline
ORG-TOPICDEVSLWWW-CO-NEW-5
THUIR-CO-NEW-36(a) Mean nDCG
0.3686
0.36820.36700.36500.35960.34050.33980.33880.33120.32940.32800.32730.32220.32050.31930.30940.28790.1626(c) Mean nERR
0.52890.52480.51290.51020.50980.50980.50520.49710.47830.47690.47470.47330.46720.45800.45410.45100.42880.4281

CENTRE: REPRODUCIBILITY RESULTS

Run name
THUIR-CO-NEW-2
THUIR-CO-NEW-1
SLWWW-CO-NEW-4
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
THUIR-CO-NEW-5
KASYS-CO-NEW-4
KASYS-CD-NEW-1
KASYS-CO-NEW-2
SLWWW-CO-NEW-5
KASYS-CD-NEW-3
THUIR-CO-NEW-3
baseline
THUIR-CO-NEW-4
KASYS-CD-NEW-5
ORG-TOPICDEVRun name
SLWWW-CO-NEW-4
SLWWW-CO-REP-1
KASYS-CO-REV-6
THUIR-CO-NEW-5
THUIR-CO-NEW-2
THUIR-CO-NEW-4
THUIR-CO-NEW-1
SLWWW-CO-NEW-3
SLWWW-CO-NEW-2
KASYS-CD-NEW-1
KASYS-CD-NEW-3
KASYS-CO-NEW-4
KASYS-CO-NEW-2
baseline
KASYS-CD-NEW-5
THUIR-CO-NEW-3
SLWWW-CO-NEW-5
ORG-TOPICDEV(b) Mean Q
0.29440.29310.28910.28900.2886
0.27180.26700.26670.25660.25480.25390.25380.25380.24940.24730.22880.20860.0857(d) Mean iRBU
0.7986
0.78400.78110.75450.75440.75100.74490.73680.73580.73510.73480.73460.73430.73270.7206
0.7166
0.71330.4526

the reproducibility measures, thus it represents a successful reproducibility attempt. This is aligned with the results in Table 9,
showing that SLWWW-CO-REP-1 performs similarly to the originalrun KASYS-CO-REV-6.
Figure 2a reports KTU at varying cut-offs. The average KTU
across topics is KTU = 0.1477 with respect to the entire run, i.e.,
1000 rank positions. Even if this value is quite low, it is higher
than those reported in other reproducibility attempts [2, 17]. Recall
that KTU is the strictest measures because it requires the same
document at each rank position for the reproduced and originalruns.
Figure 2b reports RBO at varying cut-offs. RBO for the entire
run averaged across topics is close to one: RBO = 0.9686. Notethat KTU and RBO have different trends: KTU decreases at higher
cut-offs while RBO increases (compare Figures 2a and 2b). This

Reproducibility is evaluated by comparing KASYS-CO-REV-6 as theoriginal run and SLWWW-CO-REP-1 as the reproduced run. KTU andRBO are computed at varying cut-offs thresholds. RBO is computedwith 𝜙 = 0.9, which roughly corresponds to greater weight on
the top 10 rank positions (the smaller 𝜙, the more top-heavy themeasure) [24]. RMSE and the 𝑡-test are instantiated with the same
effectiveness measures used for performance evaluation: nDCG@10,
Q@10, nERR@10 and iRBU@10, computed with the Gold relevanceassessments. All reproducibility measures are computed with therepro_eval13 library [3].
Reproducibility results are reported in Figure 2. The reproducibility run SLWWW-CO-REP-1 achieves high scores with respect to all
13 https://github.com/irgroup/repro_eval

237

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 10: Randomised Tukey HSD test results (𝐵 = 5, 000 trials) for the Gold-based results in Table 9. The runs in the left
column are statistically significantly better than those in the right column at the 5% significance level. Note that this tableomits Section (c) as none of the differences in terms of Mean nERR are statistically significant.
(a) Mean nDCG
SLWWW-CO-REP-1
ORG-TOPICDEVKASYS-CO-REV-6
ORG-TOPICDEVTHUIR-CO-NEW-2
ORG-TOPICDEVSLWWW-CO-NEW-4 ORG-TOPICDEVTHUIR-CO-NEW-1
ORG-TOPICDEVTHUIR-CO-NEW-5
ORG-TOPICDEVSLWWW-CO-NEW-2 ORG-TOPICDEVSLWWW-CO-NEW-3 ORG-TOPICDEVKASYS-CO-NEW-4
ORG-TOPICDEVKASYS-CD-NEW-1
ORG-TOPICDEVORG-TOPICDEVKASYS-CD-NEW-3
KASYS-CO-NEW-2
ORG-TOPICDEVORG-TOPICDEVTHUIR-CO-NEW-3
baseline
ORG-TOPICDEVSLWWW-CO-NEW-5 ORG-TOPICDEVTHUIR-CO-NEW-4
ORG-TOPICDEVKASYS-CD-NEW-5
ORG-TOPICDEV(b) Mean Q
THUIR-CO-NEW-2
ORG-TOPICDEVTHUIR-CO-NEW-1
ORG-TOPICDEVSLWWW-CO-NEW-4 ORG-TOPICDEVKASYS-CO-REV-6
ORG-TOPICDEVSLWWW-CO-REP-1
ORG-TOPICDEVSLWWW-CO-NEW-2 ORG-TOPICDEVSLWWW-CO-NEW-3 ORG-TOPICDEVTHUIR-CO-NEW-5
ORG-TOPICDEVKASYS-CO-NEW-4
ORG-TOPICDEVKASYS-CD-NEW-1
ORG-TOPICDEVKASYS-CO-NEW-2
ORG-TOPICDEVSLWWW-CO-NEW-5 ORG-TOPICDEVKASYS-CD-NEW-3
ORG-TOPICDEVTHUIR-CO-NEW-3
ORG-TOPICDEV
baseline
ORG-TOPICDEVTHUIR-CO-NEW-4
ORG-TOPICDEVKASYS-CD-NEW-5
ORG-TOPICDEV

happens because as the cut-off increases also the overlap between
the original and reproduced runs increases, consequently RBO scoreincreases. Conversely, when KTU considers a higher cut-off thenumber of discordant pairs increases, so KTU score decreases.
Finally, Table 2c reports RMSE scores and p-values. As for ranking measures, these results are better than those reported in otherreproducibility experiments [2, 17]. With respect to RMSE, theworst value is obtained with nERR. This might happen because
nERR is one of the most top-heavy measures and even a small error at rank position 1 or 2 can affect the measure score to a great
extent [4]. All p-values are much higher than 0.05, showing thatthe difference between the original and reproduced runs is notstatistically significant (see also discussion in Section 5).

238(d) Mean iRBUSLWWW-CO-NEW-4 ORG-TOPICDEVSLWWW-CO-REP-1
ORG-TOPICDEVKASYS-CO-REV-6
ORG-TOPICDEVTHUIR-CO-NEW-5
ORG-TOPICDEVTHUIR-CO-NEW-2
ORG-TOPICDEVTHUIR-CO-NEW-4
ORG-TOPICDEVTHUIR-CO-NEW-1
ORG-TOPICDEVSLWWW-CO-NEW-3 ORG-TOPICDEVSLWWW-CO-NEW-2 ORG-TOPICDEVKASYS-CD-NEW-1
ORG-TOPICDEVKASYS-CD-NEW-3
ORG-TOPICDEVKASYS-CO-NEW-4
ORG-TOPICDEVKASYS-CO-NEW-2
ORG-TOPICDEV
baseline
ORG-TOPICDEVKASYS-CD-NEW-5
ORG-TOPICDEVTHUIR-CO-NEW-3
ORG-TOPICDEVSLWWW-CO-NEW-5 ORG-TOPICDEVNTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 11: Official results based on the Bronze-All file (mean over the 50 WWW-4 test topics).
Run name
THUIR-CO-NEW-2
THUIR-CO-NEW-1
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-4
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-5
THUIR-CO-NEW-3
baseline
KASYS-CD-NEW-1
KASYS-CD-NEW-3
KASYS-CO-NEW-2
THUIR-CO-NEW-5
KASYS-CO-NEW-4
THUIR-CO-NEW-4
KASYS-CD-NEW-5
ORG-TOPICDEVRun name
THUIR-CO-NEW-2
THUIR-CO-NEW-1
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-4
THUIR-CO-NEW-3
ORG-TOPICDEVSLWWW-CO-NEW-5
THUIR-CO-NEW-4
baseline
KASYS-CD-NEW-3
THUIR-CO-NEW-5
KASYS-CD-NEW-1
KASYS-CO-NEW-2
KASYS-CO-NEW-4
KASYS-CD-NEW-5
0.7
0.5

RBO

Kendall's

0.6
0.4
0.3
0.2
5

10

20

50

Cut-off values

100(a) Kendall’s 𝜏 Union (KTU)

1000(a) Mean nDCG
0.62490.61110.59310.5846
0.57500.56000.54640.54100.53040.51700.51470.51300.50900.50540.50250.48140.40970.2468(c) Mean nERR
0.79670.79620.76340.75370.73300.72420.72090.70910.69770.69390.67830.6711
0.66290.65570.65190.6427
0.63840.5666
0.968
0.967
0.966
0.965
0.964
0.963
0.962
0.961

Run name
THUIR-CO-NEW-2
KASYS-CO-REV-6
THUIR-CO-NEW-1
SLWWW-CO-REP-1
SLWWW-CO-NEW-4
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-5
THUIR-CO-NEW-3
KASYS-CD-NEW-1
baseline
KASYS-CD-NEW-3
KASYS-CO-NEW-2
KASYS-CO-NEW-4
THUIR-CO-NEW-5
THUIR-CO-NEW-4
KASYS-CD-NEW-5
ORG-TOPICDEVRun name
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-2
SLWWW-CO-NEW-4
SLWWW-CO-NEW-3
THUIR-CO-NEW-1
THUIR-CO-NEW-2
THUIR-CO-NEW-3
KASYS-CD-NEW-3
baseline
KASYS-CO-NEW-4
KASYS-CD-NEW-1
KASYS-CO-NEW-2
SLWWW-CO-NEW-5
THUIR-CO-NEW-5
THUIR-CO-NEW-4
KASYS-CD-NEW-5
ORG-TOPICDEV(b) Mean Q
0.58570.57430.56910.56290.53970.5316
0.51370.51130.48530.48420.48060.47990.47330.46580.46290.44020.37390.1384(d) Mean iRBU
0.94240.93970.92440.92130.91920.91060.90280.89790.89220.89200.89120.89050.89020.88880.87930.87810.83990.6998

nDCG
Q
nERR
iRBU
5

10

20

50

Cut-off values

100(b) Rank Biased Overlap (RBO)

RMSE

p-values

0.02530.02770.03370.02710.91090.90980.9944
0.46121000(c) Reproducibility effectiveness measures

Figure 2: Reproducibility results: ranking measures KTU and RBO with varying cut-offs (Figures 2a and 2b) and reproducibility
effectiveness measures RMSE and p-values (Table 2c).

239

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 12: Randomised Tukey HSD test results (𝐵 = 5, 000 trials) for the Bronze-All-based results in Table 11(a) and (b). The runsin the left column are statistically significantly better than those in the right column at the 5% significance level.

THUIR-CO-NEW-2
THUIR-CO-NEW-1
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-4
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-5
THUIR-CO-NEW-3
baseline
KASYS-CD-NEW-1
KASYS-CD-NEW-3
KASYS-CO-NEW-2
THUIR-CO-NEW-5
KASYS-CO-NEW-4
THUIR-CO-NEW-4
KASYS-CD-NEW-5
THUIR-CO-NEW-2
KASYS-CO-REV-6
THUIR-CO-NEW-1
SLWWW-CO-REP-1
SLWWW-CO-NEW-4
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-5
THUIR-CO-NEW-3
KASYS-CD-NEW-1
baseline
KASYS-CD-NEW-3
KASYS-CO-NEW-2
KASYS-CO-NEW-4
THUIR-CO-NEW-5
THUIR-CO-NEW-4
KASYS-CD-NEW-5(a) Mean nDCG
THUIR-CO-NEW-5,KASYS-CO-NEW-4,THUIR-CO-NEW-4,KASYS-CD-NEW-5,ORG-TOPICDEVTHUIR-CO-NEW-4,KASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEV(b) Mean Q
THUIR-CO-NEW-4,KASYS-CD-NEW-5,ORG-TOPICDEVTHUIR-CO-NEW-4,KASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEV240

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 13: Randomised Tukey HSD test results (𝐵 = 5, 000 trials) for the Bronze-All-based results in Table 11(c) and (d). The runsin the left column are statistically significantly better than those in the right column at the 5% significance level.

THUIR-CO-NEW-2
THUIR-CO-NEW-1
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-2
SLWWW-CO-NEW-3
SLWWW-CO-NEW-4
KASYS-CO-REV-6
SLWWW-CO-REP-1
SLWWW-CO-NEW-2
SLWWW-CO-NEW-4
SLWWW-CO-NEW-3
THUIR-CO-NEW-1
THUIR-CO-NEW-2
THUIR-CO-NEW-3
KASYS-CD-NEW-3
baseline
KASYS-CO-NEW-4
KASYS-CD-NEW-1
KASYS-CO-NEW-2
SLWWW-CO-NEW-5
THUIR-CO-NEW-5
THUIR-CO-NEW-4
KASYS-CD-NEW-5(c) Mean nERR
KASYS-CO-NEW-2,KASYS-CO-NEW-4,KASYS-CD-NEW-5
KASYS-CO-NEW-2,KASYS-CO-NEW-4,KASYS-CD-NEW-5
KASYS-CD-NEW-5
KASYS-CD-NEW-5
KASYS-CD-NEW-5
KASYS-CD-NEW-5
KASYS-CD-NEW-5(d) Mean iRBU
KASYS-CD-NEW-5,ORG-TOPICDEVKASYS-CD-NEW-5,ORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVORG-TOPICDEVTable 14: Run ranking correlations in terms of Kendall’s 𝜏 with 95%CIs (𝑛 = 18 runs).
(a) Gold
nDCG
Q
nERR
(b) Bronze-All
nDCG
Q
nERR

Q
nERR
0.824 [0.677, 0.908] 0.627 [0.372, 0.794]
0.699 [0.477, 0.837]
QnERR
0.961 [0.924, 0.980] 0.725 [0.517, 0.852]
0.686 [0.457, 0.830](c) Gold vs. Bronze-All
nDCG 0.595 [0.327, 0.775]
Q
0.680 [0.449, 0.826]
nERR 0.327 [−0.007, 0.595]
iRBU 0.438 [0.123, 0.673]

241

iRBU
0.725 [0.517, 0.852]
0.601 [0.335, 0.778]
0.536 [0.247, 0.737]
iRBU
0.699 [0.477, 0.837]
0.712 [0.497, 0.845]
0.503 [0.204, 0.716]

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

7

CONCLUSIONS[4] Marco Ferrante, Nicola Ferro, and Maria Maistro. 2015. Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness. In The 2015
International Conference on The Theory of Information Retrieval. 21–30.
[5] Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, and Ian Soboroff. 2019.
CENTRE@CLEF 2019. In Proceedings of ECIR 2019 Part II (LNCS 11438). 283–290.
[6] Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, and Ian Soboroff. 2019.
CENTRE@CLEF: Sequel in the Systematic Reproducibility Realm. In Proceedingsof CLEF 2019 (LNCS 11696). 287–300.
[7] Nicola Ferro, Maria Maistro, Tetsuya Sakai, and Ian Soboroff. 2018. Overview of
CENTRE@CLEF 2018: a First Tale in the Systematic Reproducibility Realm. In
Proceedings of CLEF 2018 (LNCS 11018). 239–246.
[8] M. G. Kendall. 1948. Rank correlation methods. Griffin, Oxford, England.
[9] Cheng Luo, Tetsuya Sakai, Yiqun Liu, Zhicheng Dou, Chenyan Xiong, andJingfang Xu. 2017. Overview of the NTCIR-13 We Want Web Task. In
Proceedings of NTCIR-13. 394–401. http://research.nii.ac.jp/ntcir/workshop/
OnlineProceedings13/pdf/ntcir/01-NTCIR13-OV-WWW-LuoC.pdf[10] Jiaxin Mao, Tetsuya Sakai, Cheng Luo, Peng Xiao, Yiqun Liu, and Zhicheng Dou.
2019. Overview of the NTCIR-14 We Want Web Task. In Proceedings of NTCIR14. 455–467. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings14/pdf/
ntcir/01-NTCIR14-OV-WWW-MaoJ.pdf[11] Masaki Muraoka, Zhaohao Zeng, and Tetsuya Sakai. 2020. SLWWW at the
NTCIR-15 WWW-3 Task. In Proceedings of NTCIR-15. 243–246.
[12] Tetsuya Sakai. 2014. Metrics, Statistics, Tests. In PROMISE Winter School 2013:
Bridging between Information Retrieval and Databases (LNCS 8173). 116–163.
[13] Tetsuya Sakai. 2018. Laboratory Experiments in Information Retrieval: Sample
Sizes, Effect Sizes, and Statistical Power. Springer. https://link.springer.com/book/10.1007/978-981-13-1199-4[14] Tetsuya Sakai. 2019. How to Run an Evaluation Task: with a Primary Focus on
Ad Hoc Information Retrieval. In Information Retrieval Evaluation in a Changing
World: Lessons Learned from 20 Years of CLEF, Nicola Ferro and Carol Peters (Eds.).
Springer, 71–102.
[15] Tetsuya Sakai, Nicola Ferro, Ian Soboroff, Zhahao Zeng, Peng Xiao, and Maria
Maistro. 2019. Overview of the NTCIR-14 CENTRE Task. In Proceedings of NTCIR14. 494–509. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings14/pdf/
ntcir/01-NTCIR14-OV-CENTRE-SakaiT.pdf[16] Tetsuya Sakai, Sijie Tao, and Zhaohao Zeng. 2022. Relevance Assessments for Web
Search Evaluation: Should We Randomise or Prioritise the Pooled Documents?
ACM TOIS 40, 4 (2022), Article 76.
[17] Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu,
Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro, and Ian Soboroff. 2020.
Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task. In
Proceedings of NTCIR-15. 219–234.
[18] Tetsuya Sakai and Zhaohao Zeng. 2019. Which Diversity Evaluation Measures
are “Good”?. In Proceedings of ACM SIGIR 2019. 595–604.
[19] Tetsuya Sakai and Zhaohao Zeng. 2020. Retrieval Evaluation Measures that
Agree with Users’ SERP Preferences: Traditional, Preference-based, and Diversity
Measures. ACM TOIS 39, 2 (2020), Article 14.
KASYS[20] Kohei Shinden, Atsuki Maruta, and Makoto P. Kato. 2020.
at the NTCIR-15 WWW-3 Task. In Proceedings of NTCIR-15. 235–238.
https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings15/pdf/ntcir/02NTCIR15-WWW-ShindenK.pdf[21] Ian Soboroff, Nicola Ferro, Maria Maistro, and Tetsuya Sakai. 2020. Overview of
the TREC 2018 CENTRE Track. In Proceedings of TREC 2018.
[22] Yuya Ubukata, Masaki Muraoka, Sijie Tao, and Tetsuya Sakai. 2022. SLWWW at
the NTCIR-16 WWW-4 Task. In Proceedings of NTCIR-16. to appear.
[23] Kota Usuha, Kohei Shinden, and Makoto P. Kato. 2022. KASYS at the NTCIR-16
WWW-4 Task. In Proceedings of NTCIR-16. to appear.
[24] William Webber, Alistair Moffat, and Justin Zobel. 2010. A Similarity Measure
for Indefinite Rankings. ACM TOIS 4, 28 (2010), Article 20.
[25] Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible Ranking
Baselines Using Lucene. Journal of Data and Information Quality 10, 4 (2018),
Article 16.
[26] Shenghao Yang, Haitao Li, Zhumin Chu, Jingtao Zhan, Yiqun Liu, Min Zhang,and Shaoping Ma. 2022. THUIR at the NTCIR-16 WWW-4 Task. In Proceedingsof NTCIR-16. to appear.
[27] Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval. In
Proceedings of EMNLP-IJCNLP 2019. 3490–3496. https://aclanthology.org/D191352

This paper provided an overview of the NTCIR-16 We Want Web
with CENTRE (WWW-4) task. Our conclusions are as follows:• Our Gold and Bronze relevance assessments differ substantially. This is largely because while all Bronze assessments
are based on the RND (randomised) pool files, the Gold assessments are based on PRI (prioritised) pool files for one halfof the topic set. (Further details will be reported elsewhere.)
Due to the disagreements, the Gold and Bronze-All systemrankings in terms of Mean nERR are not even statisticallysignificantly correlated.
• In both Gold and Bronze-All evaluations, none of the runs
statistically significantly outperform the REV run (i.e., SOTA
from NTCIR-15). Hence we are not seeing any substantial
technological advance. However, THUIR-CO-NEW-2 [26] isquite successful in the Bronze-All evaluation in the sensethat it is the only run that managed to outperform five otherruns in terms of Mean nDCG.
• The only REP run, SLWWW-CO-REP-1 [22], is quite successful.
Its effectiveness is very similar to KASYS-CO-REV-6 [23],whose algorithm is identical to that of KASYS-E-CO-NEW-1
from the NTCIR-15 WWW-3 task [20]. Our suite of reproducibility measures also suggest that this run is more successful than previous reproducibility efforts.
Unfortunately, only the University of Tsukuba (KASYS), Waseda
University (SLWWW), and Tsinghua University (THUIR) participated in WWW-4, so we will not continue the task in its current
form. Our current plan is to propose a group-fair web search task
for NTCIR-17 by leveraging the new Chuweb21 corpus.

ACKNOWLEDGEMENTS
The WWW-4 participants (KASYS, SLWWW, THUIR) have madeour work possible. Thank you! In particular, we are indebted to
KASYS for providing a REV run, which made our discussion ofprogress and reproducibility possible. We also thank the NTCIR
chairs and staff for their constant support.
This paper was partially supported by the the European Union’s
Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No. 893667.

DISCLAIMER
Certain companies and products are identified in this paper in orderto specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by
NIST, nor is it intended to imply that the products or companies
identified are necessarily the best available for the purpose.

REFERENCES[1] Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, andEmine Yilmaz. 2008. Relevance Assessment: Are Judges Exchangeable and DoesIt Matter?. In Proceedings of ACM SIGIR 2008. 667–674.
[2] Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp
Shaer, and Ian Soboroff. 2020. How to Measure the Reproducibility of Systemoriented IR Experiments. In Proceedings of ACM SIGIR 2020. 349–358.
[3] Timo Breuer, Nicola Ferro, Maria Maistro, and Philipp Schaer. 2021. repro_eval: A
Python Interface to Reproducibility Measures of System-Oriented IR Experiments.
In Proceedings of ECIR 2021 Part II (LNCS 12657). 481–486.

242