Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan
DOI: https://doi.org/10.20736/0002001350

Fairness-based Evaluation of Conversational Search:
A Pilot Study
Tetsuya Sakai
Waseda University, Japantetsuyasakai@acm.org

ABSTRACT

2 RELATED WORK
2.1 GFR (Group Fairness and Relevance)

NTCIR-17 introduced the FairWeb-1 task, which evaluated web pagerankings in terms of both relevance and group fairness. The present
study shows how their evaluation framework can be extended forthe evaluation of multi-turn, textual conversational search systems.
By using the full test topic set of FairWeb-1 to harvest actual usersystem conversations from the New Bing and Google Bard, we
demonstrate how a series of system turns can be evaluated using
our evaluation framework, which we call GFRC (Group Fairness
and Relevance of Conversations). In addition, based on observations
from our pilot experiment, we briefly discuss a few open questions
in human-in-the-loop evaluation of conversational search in general.

1

Given 𝑀 different attribute sets (i.e., sets of groups) to consider forgroup fairness, the GFR score for evaluating a ranked list (𝐿) of
documents is defined as follows [25].
GFR(𝐿) =|𝐿 |Õ

Decay(𝐿, 𝑘) 𝑤 0 Utility(𝐿, 𝑘) +

𝑘=1

𝑀
Õ

!
𝑚

𝑤𝑚 DistrSim (𝐿, 𝑘) .

𝑚=1

(1)
Here, Decay(𝐿, 𝑘) denotes the ERR-based decay function definedover the ranked list, Utility(𝐿, 𝑘) denotes the utility of the (SearchEngine Result Page) for the group of users who abandon the SERP
at rank 𝑘, and DistrSim𝑚 (𝐿, 𝑘) is the similarity between the distribution over groups (in the 𝑚-th attribute set) achieved by the
top 𝑘 documents in the SERP and the target distribution for thesame attribute set. The weights 𝑤𝑖 (𝑖 = 0, . . . , 𝑀) balance the Utility
and DistrSim’s. At the NTCIR-17 FairWeb-1 task, movie topics (Mtopics) had 𝑀 = 2 attribute sets: RATINGS (number of ratings inIMDb) and ORIGIN (geographical region based on “country of origin” in IMDb); researcher topics (R-topics) also had 𝑀 = 2 attributesets: HINDEX (Google Scholar h-index) and GENDER (whether “he”
or “she” was used in the researcher biography, or not); YouTube
topics (Y-topics) had 𝑀 = 1 attribute set: SUBSCS (number ofsubscribers of the content uploader). Participating groups were encouraged to develop systems that provide more exposure to entities(i.e., movies, researchers, or YouTube videos) that have not received
much of it and achieve high SERP quality in terms of relevance.
Sakai et al. [25] explain that GFR is a measure of “expected user
experience” over a user population for a given information need,4and discuss its advantages over a measure used at the TREC 2022
Fair Ranking Track [8], which has been discontinued. One important advantage of GFR is that it compares distributions over ordinalgroups (such as the RATING, HINDEX and SUBSCS groups of the
FairWeb-1 task) using appropriate divergences: more specifically,
while GFR uses Jensen-Shannon Divergence (JSD) for nominalgroups (such as ORIGIN and GENDER groups of the FairWeb-1
task), it uses Normalised Match Distance (NMD) and Root Normalised Order-aware Divergence (RNOD) for ordinal groups [20].

INTRODUCTION

For the past few decades, offline web search evaluation usuallymeant “evaluating a ranked list of URLs” with measures such as
nDCG (normalised Discounted Cumulative Gain) [12] and ERR(Expected Reciprocal Rank) [6]. However, the advent of conversational search engines based on large language models (LLMs)
is rapidly changing the research landscape in search: in various
search scenarios, the user may prefer to receive direct answers froma conversational search engine rather than a list of URLs from a
traditional one. Evaluating a series of textual responses from conversational search is of utmost importance, as while their fluency
may mislead the user into thinking that their responses are trustworthy and fair, they in fact hallucinate often, and may be biased,or even be harmful: see, for example, Askell et al. [1], Liang et al.
[14], Liu et al. [16], Sakai [21].
Within the traditional ranked list evaluation paradigm, the NTCIR17 FairWeb-1 task [26]1 evaluated participating runs based not only
on relevance (for the benefit of the searcher) but also on groupfairness [7] (for the benefit of the items being ranked or their stakeholders); to this end, they employed the GFR (Group Fairness andRelevance) framework [25]. The present study shows how thisframework can be extended for the evaluation of multi-turn, textual conversational search systems. By using the full test topic set
of FairWeb-1 to harvest actual user-system conversations from the
New Bing2 and Google Bard,3 we demonstrate how a series of system turns can be evaluated using our evaluation framework, which
we call GFRC (Group Fairness and Relevance of Conversations).
In addition, based on observations from our pilot experiment, we
briefly discuss a few open questions in human-in-the-loop evaluation of conversational search in general.

2.2

S-measure and M-measure

S-measure [23], which was adopted at the NTCIR 1CLICK tasks [24],is a measure for evaluating a textual summary returned in response
to a query. One novel feature of S-measure is that it discounts thevalues of iUnits (information units or nuggets, which represent
atomic pieces of relevant information) within the summary basedon the nugget position, just like nDCG discounts the value of each

1 http://sakailab.com/fairweb1/
2 https://www.bing.com/

4 This generalises the Normalised User Utility of Sakai and Robertson [22].

3 https://bard.google.com/

5Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

document based on its rank. Let 𝑉 be the set of iUnits found within
the summary, and for each iUnit 𝑣 ∈ 𝑉 , let pos(𝑣) denote the position of 𝑣 within the summary. Moreover, let 𝑤 (𝑣) be the weightof 𝑣, which reflects its relevance grade. Then S-measure can be
expressed as follows.
𝑆-measure =1 Õ
𝑤 (𝑣) max(0, 1 − pos(𝑣)/𝐿) .
Nshould be as simple and explainable to the researchers as possible,
and our proposed method is based on this view.

3 PROPOSED EVALUATION METHOD
3.1 GFRC: A Generic Formulation
Our proposal is to combine the ideas of GFR and S-measure inorder to evaluate a series of textual system turns based on both
relevance and group fairness. Suppose that we want to evaluate a
𝑇 -round user-system textual conversation 𝐶 = (𝑈 1, 𝑆 1, . . . , 𝑈𝑇 , 𝑆𝑇 ).
Let 𝑛𝑖 𝑗 denote the 𝑗-th nugget in the 𝑖-th system turn 𝑆𝑖 ; more
specifically, we define 𝑛𝑖 𝑗 to be any substring of 𝑆𝑖 that represents a relevant piece of information in the context of the previous
turn sequence (𝑈 1, 𝑆 1, . . . , 𝑈𝑖 ) as well as the previous nuggets in 𝑆𝑖 ,i.e., (𝑛𝑖1, . . . , 𝑛𝑖 ( 𝑗−1) ). (For convenience, hereafter we denote 𝑛𝑖 𝑗 as
an element of set 𝑆𝑖 .) Furthermore, let pw(𝑛)(∈ [0, 1]) denote theposition-based weight of nugget 𝑛, where the position is defined in
the context of conversation 𝐶. (We shall introduce an instantiationof pw(𝑛) in Section 3.2.) Let 𝑔(𝑛)(∈ [0, 1]) denote the relevance
score (or gain value) of nugget 𝑛. We define the relevance-based
score of conversation 𝐶 as:

(2)

𝑣 ∈𝑉

Here, N is a normalisation factor, and 𝐿 is a parameter that reflects
user patience: at the 1CLICK tasks where Japanese summaries were
evaluated, the organisers let 𝐿 = 1, 000 based on the observationthat the average reading speed of Japanese text is around 500 characters per minute, and the assumption that the user needs to gather
information within two minutes. Thus, the value of an iUnit decreases linearly as pos(𝑣) becomes larger, and becomes nought
when pos(𝑣) > 1000.
NTCIR-11 introduced the MobileClick task [13] as a successor of
the 1CLICK task: the new task attempted to evaluate two-layeredsummaries, where the first-layer summaries contained several clickable links that represented possible intents for a given query. The
intent probabilities were regarded as the transition probabilities
from the first-layer summary to one of the second-layer summaries;
the resultant extension of S-measure was named M-measure. While
the two-layer summary framework resembles a two-round conversation that branches out depending on the previous turn, theseprevious NTCIR tasks considered the relevance of information only,
unlike the present study.

2.3

𝑇

𝑅(𝐶) =1 Õ Õ
pw(𝑛𝑖 𝑗 )𝑔(𝑛𝑖 𝑗 ) ,
N 𝑖=1

(3)

𝑛𝑖 𝑗 ∈𝑆𝑖

where N is a normalisation factor.
Next, we describe how the same conversation 𝐶 can be evaluatedin terms of group fairness, given 𝑀 attribute sets and a target
distribution (a probability mass function, to be more specific) 𝐷𝑚∗
for each 𝑚(= 1, . . . , 𝑀). For every system turn 𝑆 that contains a
relevant nugget, we first compute an achieved distribution 𝐷𝑚 (𝑆)using one of the following two possible methods. The independent
distribution method computes 𝐷𝑚 (𝑆) based solely on the groupmemberships of relevant nuggets contained in 𝑆; the cumulative
distribution method computes 𝐷𝑚 (𝑆) based not only on relevant
nuggets in 𝑆 but also on relevant nuggets observed in the previous
system turns.5 On the other hand, for any system turn that does
not contain a relevant nugget, we can either ignore it, or treat it as
if its achieved distribution is uniform (because if the turn does not
mention any relevant entity, it is not introducing any bias towards
any entity group and therefore “fair”).
Let PW(𝑆) denote the position-based weight of turn 𝑆; we shallinstantiate it later. The Group Fairness (GF) score of conversation
𝐶 can be computed as:

Other Related Work

The TREC 2022 Conversational Assistance Track [17] evaluatedmulti-turn conversational search using a topic tree structure; while
the conversations are evaluated in terms of relevance, conciseness,
and naturalness, they did not consider group fairness. The track
has been discontinued.
Inspired by the aforementioned S-measure, our proposed framework uses word counts as the basis for discounting a value of a
relevant piece of information (i.e., a nugget). In the instantiationof our framework that is discussed in the present study, all words
from a conversation (including those from user turns) contributeto the word count. This means that, if the user enters a long query,that is considered a large cost, and the values of systems’ nuggets
decrease accordingly. This mechanism is related to the discussionof typing cost or cost of querying in the economic interaction modelsof Azzopardi and Zuccon [3]. Earlier studies that consider the costof querying (in terms of time spent) include Baskaya et al. [4] andAzzopardi et al. [2].
Regarding the evaluation of LLM-based systems, one obvious
approach would be to build an LLM-based evaluator. However, if ablack box LLM is evaluated with another black box LLM, it is ouropinion that we may not be able to learn much: the two may bebased on similar training data with similar algorithms and therefore
the evaluator may overrate the system being evaluated; moreover, it
may be difficult for an end-to-end LLM-based evaluator to explain to
us why the system is good/bad and where exactly the problem lies.
See also Faggioli et al. [9] and a recent Dagstuhl Seminar report [5,p.47] for related discussions. We argue that evaluation methods

GF(𝐶) =

𝑇
𝑀
Õ
1 Õ
PW(𝑆
)
𝑤𝑚 DistrSim𝑚 (𝐷𝑚 (𝑆𝑖 ) ∥ 𝐷𝑚𝑖∗ ) , (4)N ′ 𝑖=1𝑚=1where N ′ is a normalisation factor and 𝑤𝑚 is a weight assigned toÍ𝑀𝑤𝑚 = 1. Following GFR, we use JSD
the 𝑚-th attribute set s.t. 𝑚=1
for computing the DistrSim function if the attribute set contains
nominal groups, and we use either NMD or RNOD if the attribute
5 The latter approach resembles the GFR framework [25], which computes an achieved

distribution for each user group that is assumed to abandon the ranked list at a particular
relevant document at rank 𝑟 ; the group memberships of all relevant entities in the
top 𝑟 documents contribute to the achieved distribution, which means that the groupmemberships of relevant entities near the top ranks contribute relatively heavily tothe overall GFR score.

6

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

set contains ordinal groups [25]. Note that the DistrSim value isobtained as one minus a divergence such as JSD.
As a “quick summary” measure for ranking systems, the following combined measure may be of some use:
GFR(𝐶) = 𝛼𝑅(𝑐) + (1 − 𝛼)𝐺𝐹 (𝐶) ,U1

U2

Based onFairWeb-1
test topic

(5)

S2

Can you namea few more?
Adhere to the
output format
that I
specified…

Please list up…
Format: …

where the 𝛼 is a parameter that balances relevance and group fairness. In practice, however, we recommend to report R and GF scores
separately and to visualise the relationship between the two.

3.2

S1

Dialoguebreakdown

pw(n)=1 when WC(n)=1

GFRC: An Instantiation

pw(n)=0 when WC(n)=L+1

In this section, we discuss a specific and practical instantiation of
GFRC.
First, we provide a specific definition of pw(𝑛), the position-based
nugget weight. Unlike the NTCIR 1CLICK tasks which primarily
dealt with Japanese summaries, we discuss evaluating English textual conversations, and therefore consider word count-based positions rather than character count-based ones. Let us assume thatthe user is willing to spend up to 5 minutes to satisfy a particular
information need; furthermore, we assume that the average reading
speed of the user is 250 words per minute.6 It then follows thatthe user is willing to read up to 𝐿 = 1, 250 words. For any word 𝑤
from conversation 𝐶, let wc(𝑤) denote its word count: for example,
for the first word in user turn 𝑈 1 , 𝑤𝑐 = 1. Furthermore, for any
relevant nugget 𝑛 within a system turn, we define its word countWC(𝑛) as the word count of the last word that 𝑛 corresponds to.
Then we instantiate the position-based nugget weight in Eq. 3 as:

Figure 1: A two-round conversation protocol used in the pilot experiment.

consider a simpler option: let S be the set of system turns excluding those that do not contain any relevant entity, and simply let
PW(𝑆𝑖 ) = 1 iff 𝑆𝑖 ∈ S, with N ′ = |S|. That is, we simply average
the DistrSim’s over relevant system turns. Similarly, when considering 𝑀 different attribute sets in Eq. 4, we simply average theDistrSim’s across them.
In summary, our instantiations of 𝑅(𝐶) (Eq. 3) and GF(𝐶) (Eq. 4)
are:
𝑅(𝐶) =

WC(𝑛) − 1),(6)𝐿
so that the nugget weight linearly decreases as the conversation
proceeds, and any nugget beyond the word count limit of 1,250 willbe ignored. The accompanying normalisation factor N (See Eq. 3)
can then be given by:

𝐿 
Õ
𝑙 −1
𝐿+1
N=
1−
.
(7)
=
𝐿
2
pw(𝑛) = max(0, 1 −𝑇WC(𝑛𝑖 𝑗 )2 Õ Õ
max(0, 1 −)𝑔(𝑛𝑖 𝑗 ) ,𝐿 + 1 𝑖=1
𝐿

(8)

𝑛𝑖 𝑗 ∈𝑆𝑖𝑀

1 Õ 𝑚
GF (𝐶) ,
𝑚 𝑚=1

(9)

1 Õ
DistrSim𝑚 (𝐷𝑚 (𝑆𝑖 ) ∥ 𝐷𝑚
∗ ).
|S|

(10)

GF(𝐶) =
where
GF𝑚 (𝐶) =

𝑆𝑖 ∈S

𝑙=1

This is a “hard” normalisation factor, which represents a practicallyunattainable situation where every word in conversation 𝐶 represents a relevant nugget. Nevertheless, this can be applied if we do
not want the score to exceed 1.7
S-measure, which was designed for evaluating textual summaries,
used a “softer” normalisation factor based on a “minimal” summary(pseudo minimal output [23]), which is analogous to the ideal ranked
list of nDCG. However, this approach assumes a recall base ofnuggets, and is probably not suitable for conversational searchwhere it is often not possible to enumerate all relevant nuggets forevaluation.
Next, we instantiate the position-based weight for turn 𝑆 to implement Eq. 4. While it is possible to make the weight actually
position-aware by letting PW(𝑆) a function of pw(𝑛)(𝑛 ∈ 𝑆), e.g.,
the maximum nugget weight within 𝑆, in the present study, weor GF(𝐶) = 0 if S = 𝜙 (i.e., none of the system turns are relevant).

4 A PILOT EXPERIMENT
4.1 Task and Topics
To illustrate how GFRC scores can be computed from user-system
conversations, we conducted a pilot experiment using the entire 45
topics from the NTCIR-17 FairWeb-1 task, which we downloaded
from the aforementioned FairWeb-1 website. The test topic set
contains 15 M-topics, 15 R-topics, and 15 Y-topics.
Figure 1 shows the two-round (𝑇 = 2) conversation protocol weused to harvest textual interactions with a given conversational
search system. (The bottom part of the figure shows a conceptual
diagram of the linearly decaying position-based nugget weights.)
The author of this paper manually interacted with conversational
search systems on a web browser and recorded all textual exchanges.
User Turn 1 (𝑈 1 ) asks the system to list entities according to a
FairWeb-1 topic. For example, 𝑈 1 for Topic R001 was:

6 The New Bing said (as of September 7, 2023): “According to a meta-analysis of 190

studies on reading speed, the average silent reading speed for an adult person is 238
words per minute (WPM) for non-fiction, while the average reading speed for fiction
is 260 WPM”; Google Bard returned the same WPM numbers (same day); we havechecked multiple websites to see that these answers are not hallucinations.
7 An alternative would be score standardisation [19, 27, 28].

Please list up researchers who have published at least
one paper (any track) at the CHIIR conference.
Format: researcher name, URL

7

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan“x o”: (shown in yellow) In response to 𝑈 2 : “Adhere to the
output format that I specified...” the system managed to improve the response format.
“x x”: (shown in orange) In response to 𝑈 2 : “Adhere to the
output format that I specified...” the system still
failed.
“x -”: (shown in grey) 𝑆 1 caused a dialogue breakdown (e.g.,
the aforementioned case with G1 for M001).

This is a form of zero-shot prompting.8 As the figure indicates, System Turn 1 (𝑆 1 ) may cause a dialogue breakdown [11]: the user
is unable to continue the conversation due to highly inadequate
responses such as:
I'm a text-based AI, and that is outside of my capabilities.

Otherwise, the same author entered 𝑈 2 , which was either
Can you name a few more?

or one of the following depending on the topic type, as systems
often fail to follow the output format specified in 𝑈 1 :At the bottom of each figure, some simple statistics are shown;
they are not the focus of this paper. In each column, “%useful” isthe proportion of “o” responses; In the columns for Turn 2, we have“ave%useful” which simply averages the “%useful” over the twoturns; we also have “%recovery,” defined as: “Of all topics for which
the system’s first turn was unsuccessful, what is the proportion thatthe system managed to return a useful second turn?” For example,
in Figure 2, G2 has 14 unsuccessful 𝑆 1 ’s, but recovered with 𝑆 2 for
9 of the topics, and therefore %recovery= 9/14 = 64.3%.
From Figures 2-4, the following observations can be made.

Adhere to the output format I specified:"Format: movie title, IMDb URL"
Adhere to the output format I specified:"Format: researcher name, URL"
Adhere to the output format I specified:"Format: video title, youtube URL"

The entire conversations that we harvested in this way can be foundin our supplementary material package.9

4.2

Systems• First and foremost, the behaviour of each system is vastlydifferent across the two trials, even though they were testedon the same day. Moreover, once they start to perform inadequately, they tend to continue to do so. For example, compare
B1 and B2 in Figure 2 (M-topics), and G1 and G2 in Figure 3(R-topics); in both cases, the second trial was a disaster, eventhough the first was not as poor. This extremely unstable
nature of LLM-based conversational search systems posesa substantial challenge in terms of evaluation: the resultsof a small-scale experiment such as the one reported in thepresent study (let alone a few cherry-picked anecdotes wesometimes see in recent LLM papers!) are far too unreliable
and unlikely to be generalisable.
• LLMs are known to hallucinate and, not surprisingly, theNew Bing and Google Bard are no exceptions. For example, as indicated in Figure 3 footnotes 1 and 3, both of thesesystems hallucinated about the AIRS (Asia Information Retrieval Societies) conference: in fact, the final AIRS conference (not directly related to SIGIR) took place in 2019,10 , andinstead the SIGIR-AP (Asia-Pacific) conference was launchedin 2023.11 Moreover, the URLs returned by Google Bard were
often incorrect: for example, as we shall see later, most of
the IMDb URLs that it returned for Topic M002 (time travel
movies) were those for wrong movies.
• Google Bard behaved in the “o x” (light green) pattern very
often (for R-topics and Y-topics), while this never happened
with Bing in our experiment. Google’s 𝑆 2 often did not makesense: even though 𝑆 1 contained some seemingly relevant
URLs and therefore the system seems capable of performing
the task to some extent, when asked for “a few more,” 𝑆 2
was often “I’m unable to help you with that, as I’m
only a language model and don’t have the necessary
information or abilities.” or something similar.
• Interestingly, during the conversation for R012 (about SIGIR) with Bing (Trial 2), in response to “Adhere to the
output format I specified: "Format: researcher

On September 2, 2023, the author of this paper followed the above
protocol and interacted with both the New Bing and Google Bard.
For each topic type, the topics were tested in the original order,
and both systems were tested in parallel. For example, Topic M001
was tested with the New Bing and then with Google Bard; thenthe author moved on to Topic M002, and so on. Moreover, as LLMbased systems behave nondeterministically, after completing theconversations for all 45 topics, we repeated the same experiment.
Thus, we had two trials with the New Bing (denoted by B1 and B2),
and two trials with the Google Bard (denoted by G1 and G2). Notethat we tested the topics in the same order in the two trials, as we
wanted to avoid confounding the effect of topic ordering with thatof nondeterministic responses. As we shall report in Section 5, wehave evidence that a previous topic can actually affect the system
response for the current topic.

5

SYSTEM RESPONSE OVERVIEW

Figures 2-4 visualise the outcomes of the two-round conversations
with the New Bing and Google Bard using the M-topics, R-topics,
and Y-topics from FairWeb-1. In each cell, a “o” means that thesystem turn “looks” useful to some extent; that is, it provides some
URLs as requested (although they may not necessarily be relevant
as we shall discuss later). A “x” means that the system turn is notuseful: it either does not contain useful URLs (for example, for M001,
B1 𝑆 1 lists movies without providing IMDb URLs), or it refuses theuser request. For example, for M001, G1 𝑆 1 says:
I'm unable to help you with that, as I'm only a language modeland don't have the necessary information or abilities.

A “-” indicates a dialogue breakdown (for example, for M001, G1
𝑆 2 is a “-” due to 𝑆 1 shown above.) As the three figures show, all of
the following patterns were observed in the experiment:“o o”: (shown in green) Both turns returned seemingly relevantURLs;“o x”: (shown in light green) In response to 𝑈 2 : “Can you namea few more?” the system failed to respond adequately.
8 https://www.promptingguide.ai/

10 https://link.springer.com/book/10.1007/978-3-030-42835-8

9 http://waseda.box.com/evia2023pack

11 http://www.sigir-ap.org/sigir-ap-2023/

8

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

S1

S2

S1

S2

S1

S2

S1

S2

Figure 2: Overview of system responses for the M topics. B1, G1 etc. mean “Bing Trial 1” and “Google Trial1” and so on.
“o” means that the system turn “looks” useful; “x” means that it is not useful;“-” means that there was no second system turn due to dialogue breakdown.
S1

S2

S1

S2

S1

S2

S1

S2

Figure 3: Overview of system responses for the R topics. B1, G1 etc. mean “Bing Trial 1” and “Google Trial1” and so on.
“o” means that the system turn “looks” useful; “x” means that it is not useful;“-” means that there was no second system turn due to dialogue breakdown.
*1: contains an obvious hallucination which caused a dialogue breakdown: “I found the proceedings of the AIRS 2022 conference”*2: confusion with the previous topic “Here are some researchers who have published at least one paper (any track) at the KDD conference”*3: contains an obvious hallucination which caused a dialogue breakdown: “the AIRS conference has been renamed to the ACM SIGIR Conference on Researchand Development in Information Retrieval (SIGIR) since 2019”*4: the 2nd list has an overlap with the 1st list

name, URL””, the system responded with “Here are someresearchers who have published at least one paper(any track) at the KDD conference...” (Figure 3 footnote 2). This is because the previous topic discussed was
R011 (about KDD). Hence previous conversations do seemto affect the current response in some cases, despite thefact that the systems kept ignoring the same output format

instructions in many conversations.12 Because the system
responses are dependent on previous context, this adds afurther challenge to the evaluation of conversational search:
the sampled system turns are clearly not independent of one
another.
12 As of September 2, 2023, the New Bing forced the user to “start a new topic” after 30

system turns.

9

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

S1

S2

S1

S2

S1

S2

S1

S2

Figure 4: Overview of system responses for the Y topics. B1, G1 etc. mean “Bing Trial 1” and “Google Trial1” and so on.
“o” means that the system turn “looks” useful; “x” means that it is not useful;“-” means that there was no second system turn due to dialogue breakdown.
*1: the 2nd list has an overlap with the 1st list
*2: system suddenly started apologising in Japanese*3: contains duplicate URLs with different titles*4 the same URL is repeated 5 times with different titles*5 same as *4, and the URL is the same as the one in Turn 1*6 the same URL is repeated 3 times with different titles*7 experimenter failed to record the response, but probably it was “I’m just a language model” etc., i.e., an “x.”countries:13 hence we can demonstrate how group fairness
can be quantified in such evaluation settings;• For M002, both the New Bing and Google Bard managed toreturn results that are not useless;• The author of this paper was the Gold annotator of thisFairWeb-1 topic; hence the author gets to decide whichmovies are relevant or not to his own information need.

• Even when both 𝑆 1 and 𝑆 2 looked useful, for R-topics andY-topics (especially the latter), the URLs returned sometimes
had an overlap across the two turns, and/or had duplicate
URLs within a single list. We argue that such duplicatesshould not be rewarded when evaluating the system turns: it
seems sensible to treat all duplicate entities as nonrelevant, as
was proposed at NTCIR-4 in the context of factoid question
answering evaluation where only one answer string fromeach equivalence class of relevant answers was considered
relevant in a ranked answer list [18].
The discussion above demonstrates that current LLM-based conversational search engines have a lot of room for improvement, andalso that evaluating them in a reliable manner is highly challenging.
We leave the grand challenge for future work; nevertheless, as one
small piece that will hopefully contribute to the above challenge,
we henceforth discuss a case study in which we apply our proposed
evaluation framework to a topic from the above experiment.

6

6.1

Gold Annotation

The present author examined two text files that contained all the
textual interactions between himself and the system (either the
New Bing or Google Bard, Trial 1). On September 5, 2023 (i.e., 3days after we harvested the conversations), the author examined
the texts and annotated them by inserting lines after each entity
returned by the system. Figure 5 shows the results: the annotations
are shown in red. For example, in the New Bing’s Turn 1 (𝑆 1 ), theIMDb URL for “Back to the Future” is followed by the following
pieces of information:

A CASE STUDY• This entity is L2-relevant (i.e., highly relevant) to the information need;• On the IMDb page, the movie currently has 1.3M ratings
and therefore falls into Group 4 (i.e., movies with 1M user
ratings or more) of the RATINGS attribute set;This section demonstrates how our measures can be computedfrom real conversational search data. As an example, we chose the
Trial 1 results of B1 and G1 for Topic M002 “time travel movies”(highlighted in red in Figure 2) for the following reasons:• M-topics are the only ones with two attribute sets where oneof them allows soft group membership: one movie may belong
to multiple ORIGIN groups (i.e., geographic regions) if its
country of origin field on the IMDB page mentions multiple

13 The mapping from country names to eight geographic regions is also documented on

http://sakailab.com/fairweb1/.

10

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan##### M002### U1
Please list up sci-fi movies that feature time travel, listed in IMDb.
Format: movie title, iMDB URL### S1 x##### M002### U1
Please list up sci-fi movies that feature time travel, listed in IMDb.
Format: movie title, iMDB URL### S1 oHere are some sci-fi movies that feature time travel, listed in IMDb:
Back to the Future (1985): https://www.imdb.com/title/tt0088763/#L2
wc= 35#RATINGS: (0,0,0,1) #1.3M#ORIGIN: (0,1,0,0,0,0,0,0) #United States
The Terminator (1984): https://www.imdb.com/title/tt0088247/#L2
wc= 39#RATINGS: (0,0,1,0) #898K#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States
Groundhog Day (1993): https://www.imdb.com/title/tt0107048/#L2
wc= 43#RATINGS: (0,0,1,0) #661K#ORIGIN: (0,1,0,0,0,0,0,0) #United States
Interstellar (2014): https://www.imdb.com/title/tt0816692/#L1
wc= 46#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
The Time Machine (1960): https://www.imdb.com/title/tt0054387/#L2
wc= 51#RATINGS: (0,0,1,0) #43K
#ORIGIN: (0,1,0,0,0,0,0,0) #United States
I hope this helps! Let me know if there’s anything else I can help you with.
### U2Can you name a few more?
### S2 o
Sure! Here are some more sci-fi movies that feature time travel, listed in IMDb:
12 Monkeys (1995): https://www.imdb.com/title/tt0114746/#L2
wc= 91#RATINGS: (0,0,1,0) #634K#ORIGIN: (0,1,0,0,0,0,0,0) #United States
The Butterfly Effect (2004): https://www.imdb.com/title/tt0289879/#L2#RATINGS: (0,0,1,0) #508K
wc= 96#ORIGIN: (0,1,0,0,0,0,0,0) #United States,Canada
Looper (2012): https://www.imdb.com/title/tt1276104/#L2
wc= 99#RATINGS: (0,0,1,0) #592K#ORIGIN: (0,1/2,0,1/2,0,0,0,0) #United States,China
Edge of Tomorrow (2014): https://www.imdb.com/title/tt1631867/#L2
wc= 104#RATINGS: (0,0,1,0) #711K#ORIGIN: (0,1,0,0,0,0,0,0) #United States,Canada
Predestination (2014): https://www.imdb.com/title/tt2397535/#L2#RATINGS: (0,0,1,0) #294K
wc= 107#ORIGIN: (0,1/2,0,0,0,0,0,1/2)#Australia,United States
I hope this helps! Let me know if there’s anything else I can help you with.

[A 452-word response that does not follow the above format]### U2Adhere to the output format I specified: "Format: movie title, iMDB URL"### S2 oSure, here are the sci-fi movies that feature time travel, listed in IMDb, in the specified format:
Movie title IMDb URL
Back to the Future (1985) https://www.imdb.com/title/tt0088763/#L2#RATINGS: (0,0,0,1) #1.3M
wc= 506#ORIGIN: (0,1,0,0,0,0,0,0) #United States
Back to the Future (1985) movie posterOpens in a new window
www.imdb.com
Back to the Future (1985) movie poster
Pulp Fiction (1994) https://www.imdb.com/title/tt0110912/#L0
Pulp Fiction (1994) movie posterOpens in a new window
www.originalfilmart.com
Pulp Fiction (1994) movie poster
Looper (2012)https://www.imdb.com/title/tt1853739/#L0 # wrong URL "You're Next"
Looper (2012) movie posterOpens in a new window
www.imdb.com
Looper (2012) movie poster
Interstellar (2014) https://www.imdb.com/title/tt0816692/#L1
wc= 560#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
Interstellar (2014) movie posterOpens in a new window
www.imdb.com
Interstellar (2014) movie poster
Tenet (2020)https://www.imdb.com/title/tt10375988/#L0 # wrong URL "The Eze-Nwanyi Kate Ajike Love Show"
Tenet (2020) movie posterOpens in a new window
www.imdb.com
Tenet (2020) movie poster
Doctor Strange (2016)
https://www.imdb.com/title/tt3498496/#L0 # wrong URL "Michigan Pheasant"Doctor Strange (2016) movie posterOpens in a new window
www.imdb.com
Doctor Strange (2016) movie poster
The Terminator (1984)
https://www.imdb.com/title/tt0089531/#L0 # wrong URL "Madame Deborah"
Terminator (1984) movie posterOpens in a new window
www.imdb.com
Terminator (1984) movie poster
The Butterfly Effect (2004)
https://www.imdb.com/title/tt0322997/#L0 # wrong URL "Cruiskeen Lawn"
Butterfly Effect (2004) movie posterOpens in a new window
www.imdb.com
Butterfly Effect (2004) movie poster
Primer (2004)
https://www.imdb.com/title/tt0389895/#L0 # wrong URL "Crystal Force 2: Dark Angel"
Primer (2004) movie posterOpens in a new window
www.imdb.com
Primer (2004) movie poster
12 Monkeys (1995) https://www.imdb.com/title/tt0112389/#L0 wrong URL "The Thief and the Cobbler"
12 Monkeys (1995) movie posterOpens in a new window
www.imdb.com
12 Monkeys (1995) movie poster
I hope this is more helpful!

Figure 5: Bing Trial 1 (left) vs. Google Trial 1 (right) for Topic M002 (time travel movies) with relevance and group membership
annotations, with word counts.
• On the IMDb page, United States is named as the countryof origin of this movie; hence this movie belongs solely toGroup 2 (America) of the ORIGIN attribute set.
In Figure 5, word count information is also provided in balloons.
For example, the word count for the “Back to the Future” URL is 35:
that is, this is the 35th word in this conversation. Recall that when
we compute the word count, words from the user turns are includedin the present study: that is, if the user enters many words, thatis considered as a cost, and the system effectiveness is discounted
accordingly.

6.2

be observed that while the New Bing managed to return 10 relevant
entities (with “Interstellar” considered by the Gold annotator to
be an L1-relevant entity unlike the other movies), Google Bard
managed to return only 2: many of the IMDb URLs it returned wereincorrect. Note that the rightmost column (pw(𝑛)𝑔(𝑛)) is analogous
to the discounted gain of nDCG: for example, while both Bing and
Google return “Back to the Future” as the first L2-relevant entity intheir responses, the pw(𝑛)𝑔(𝑛) for Google is lower (1.9456 for Bing
but 1.1920 for Google). This is largely because, as Figure 5 (right)
shows, Google’s first turn did not contain any relevant nuggets andwasted as many as 452 words. As a result, the word count for the
IMDb URL of the “Back to the Future” nugget is 506. According to
Eq. 8 with 𝐿 = 1, 250, the R score for Bing is 0.0143 while that for
Google is only 0.0014 (i.e., only about 9.8% of the R score for Bing).

Measure Computation

Table 1 shows how the R scores are computed for the conversations
shown in Figure 5 based on Eq. 8; here, L2-relevant and L1-relevant
entities are mapped to gain values of 1 and 0.5, respectively. It can

11Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

Table 1: Computing the R scores for the conversations
shown in Figure 5.
movie (Nugget 𝑛)

pw(𝑛) 𝑔(𝑛)(a) New Bing
Back to the Future
0.9728
1
The Terminator
0.9696
1
Groundhog Day
0.9664
1
Interstellar
0.96400.5
1
The Time Machine 0.9600
12 Monkeys
0.9280
1
The Butterfly Effect 0.9240
1
Looper
0.9216
1
Edge of Tomorrow
0.91761
0.91521
Predestination
𝑅(b) Google Bard
Back to the Future
0.5960
1
Interstellar
0.55280.5
𝑅America (mapped from United States and Canada), while acknowledging some presence of Europe (mapped from United Kingdom).
As the groups (i.e., regions) are nominal this time, the DistrSm values are obtained using JSD. It can be observed that, even in termsof ORIGIN-based GF scores, Google slightly underperforms Bing:
its score is about 95.8% of that of Bing (0.4303 vs 0.4493). This reflects the fact that Google only covers Groups 2 and 6 (America andEurope), while Bing covers Groups 4 and 8 (Asia and Oceania) in
addition, as can be verified from Figure 5.
In summary, in this particular example, Bing outperforms Google
in terms of all three measures: the R (relevance) score, the RATINGSbased GF score, and the ORIGIN-based GF score. This example
suggests that our framework, which extends the GFR framework forevaluating ranked lists, may be useful for improving conversational
search engine responses in terms of relevance and group fairness.

pw(𝑛)𝑔(𝑛)0.97280.9696
0.96640.48200.96000.92800.92400.92160.91760.91520.01437
0.59600.27640.0014

CONCLUSIONS

We have demonstrated how the GFR framework used in the NTCIR17 FairWeb-1 task can be extended for evaluating a series of textual
system turns in conversational search. By using the full test topic
set of FairWeb-1 to harvest actual user-system conversations fromthe New Bing and Google Bard, we demonstrated how a series ofsystem turns can be evaluated using our GFRC (Group Fairness andRelevance of Conversations) framework.
With the rapid progress of LLM-based conversational searchsystems and the various social problems that accompany it (e.g.,
hallucinations and biases), we argue that evaluating a sequenceof system turns is becoming at least as important as evaluating aranked list of documents, to put it mildly. Hence, the NTCIR FairWeb
organisers are currently considering a new conversational subtask
for NTCIR-18 along with the existing web page ranking task, so thatgroup fairness can be studied by looking across these two paradigms.
From the participants’ point of view, can the same algorithm beused for group fair ranking and group fair conversations? From theorganisers’ point of view, how are GFR (for rankings) and GFRC (forconversations) related to each other, and how can conversation trees(i.e., turns that branch out) [17] be obtained so that turn transition
probabilities can be incorporated into GFRC?
Our pilot experiment also exemplified some known challenges
in evaluating conversational search. The following are a few open
questions for human-in-the-loop conversational search evaluationthat we have observed.

Note also how the relevant entities from multiple turns aretreated seamlessly in the R score computation: for example, Figure 5(left) shows that 5 relevant entities were returned in System Turn 1(𝑆 1 ) and another 5 were returned in System Turn 2 (𝑆 2 ); but from
Table 1 it is clear that the R score treats all of these entities just as
those that lie on top of a single slope (See Figure 1 bottom).
Table 2 shows how the GF scores are computed for the conversations shown in Figure 5 based on Eq. 10. First, let us discuss theleft part of the table that computes GF scores for the RATINGS
attribute set (with 4 ordinal groups, where Group 4 represents
movies with over 1M ratings). From Figure 5 (left), the group membership vectors for the 5 relevant entities returned by Bing in 𝑆 1are (0, 0, 0, 1), (0, 0, 1, 0), (0, 0, 1, 0), (0, 0, 0, 1), (0, 0, 1.0). By averaging them we obtain the achieved distribution for 𝑆 1 : (0, 0, 0.6, 0.4)
as shown in Table 2 (left). By comparing this achieved distribution with a uniform gold distribution over the 4 groups in terms of
RNOD,14 the DistrSim value is 0.6773 for 𝑆 1 . Similarly, the achieved
distribution of 𝑆 2 for Bing is (0, 0, 1, 0) and the DistrSim in terms of
RNOD is 0.4796. Finally, by applying Eq. 10, we obtain a RATINGSbased GF score of 0.5785 for Bing. On the other hand, as only 𝑆 2contains relevant entities in the Google results, from Eq. 10, only
the DistrSim value for 𝑆 2 contributes to the RATINGS-based GF
score for Google, which amounts to 0.4049 (i.e., only about 70.0% of
the score for Bing). It can be observed that Google underperforms
Bing because it is more heavily biased towards “famous” movies(i.e., those with many ratings).
Finally, let us discuss the right part of Table 2 that computes GF
scores for the ORIGIN attribute set (with 8 nominal groups representing geographic regions). Again, the achieved distributions forthe system turns are computed by averaging the group membership
vectors shown in Figure 5: for example, the distribution for Bing’s 𝑆 1(0, 0.8333, 0, 0, 0, 0.1667, 0, 0) means that it is heavily biased towards• Given the highly unstable and context-dependent natureof the system turns based on LLMs, how can we conduct
reliable evaluation with appropriate sampling methods andsample sizes?
• How can we sample and evaluate worst-case situations (rather
than typical ones represented by mean scores etc.), so thatany possible harm on users can be detected in a timely manner?
• What is the appropriate methodology to combine humanin-the-loop evaluations (which we cannot do without, assatisfying and protecting the users is our goal) and user simulations [10, 15, 29] which enable large-scale evaluations andhence may enable a better coverage of the above-mentioned
worst-case scenarios?

14 NMD may be used instead as the groups are ordinal.

12

Proceedings of the Tenth International Workshop on Evaluating Information Access (EVIA 2023), a Satellite Workshop of the NTCIR-17 Conference, December 12-15, 2023, Tokyo, Japan

Table 2: Computing the GF scores for the conversations shown in Figure 5.

1

𝐷 RATINGS (𝑆𝑖 )
2
3
4

𝑆1
𝑆2

0
0

0
0

0.6 0.4
1
0
GFRATINGS

𝑆1
𝑆2

0

0

0

1

GFRATINGS

DistrSim(RNOD)
0.67730.4796
0.57850.40490.4049

𝐷 ORIGIN (𝑆𝑖 )
2 3
4 5
6
(a) New Bing
0 0.8333 0
0 0 0.1667
0
0.8 0 0.1 0
01(b) Google Bard
- 0 0.8333 00

REFERENCES

0

0.1667

8

DistrSim
(JSD)

0
0
0 0.1
GFORIGIN

0.43030.46820.4493

0
0
GFORIGIN

0.43030.4303

7

Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul,
Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter
Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya
Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary,
William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023.
Holistic Evaluation of Language Models. arXiv:2211.09110 [cs.CL][15] Aldo Lipani, Ben Carterette, and Emine Yilmaz. 2021. How Am I Doing?: Evaluating Conversational Search Systems Offline. ACM TOIS 39, 4, Article 51 (2021).
[16] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao
Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy
LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment.
arXiv:2308.05374 [cs.AI][17] Paul Owoicho, Jeffrey Dalton, Mohammad Aliannejadi, Leif Azzopardi, Johanne R.
Trippas4, and Svitlana Vakulenko. 2023. TREC CAsT 2022: Going Beyond User
Ask and System Retrieve with Initiative and Response Generation. In NIST Special
Publication 500-338: The Thirty-First Text REtrieval Conference Proceedings (TREC
2022) (Virtual Event). NIST.
[18] Tetsuya Sakai. 2004. New Performance Metrics based on Multigrade Relevance: Their Application to Question Answering. In Working Notes of NTCIR4. https://research.nii.ac.jp/ntcir/ntcir-ws4/NTCIR4-WN/OPEN/OPENSUB_
Tetsuya_Sakai.pdf[19] Tetsuya Sakai. 2016. A Simple and Effective Approach to Score Standardisation.
In Proceedings of ACM ICTIR 2016 (Newark, Delaware, USA). ACM, 95–104.
[20] Tetsuya Sakai. 2020. Evaluating Evaluation Measures for Ordinal Classification
and Ordinal Quantification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long Papers) (Online). Association
for Computational Linguistics, 2759–2769. https://aclanthology.org/2021.acllong.214.pdf[21] Tetsuya Sakai. 2023. SWAN: A Generic Framework for Auditing Textual Conversational Systems. arXiv:2305.08290 [cs.IR][22] Tetsuya Sakai and Noriko Kando. 2008. Modelling A User Population for
Designing Information Retrieval Metrics. In Proceedings of EVIA 2008. 30–
41. https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings7/pdf/EVIA2008/
07-EVIA2008-SakaiT.pdf[23] Tetsuya Sakai, Makoto P. Kato, and Young-In Song. 2011. Click the Search
Button and Be Happy: Evaluating Direct and Immediate Information Access. In
Proceedings of CIKM 2011 (Glasgow, Scotland, UK). ACM, 621–630.
[24] Tetsuya Sakai, Makoto P. Kato, and Young-In Song. 2011. Overview of NTCIR-9
1CLICK. In Proceedings of NTCIR-9 (Tokyo, Japan). 180–201.
[25] Tetsuya Sakai, Jin Young Kim, and Inho Kang. 2023. A Versatile Framework for
Evaluating Ranked Lists in terms of Group Fairness and Relevance. ACM TOIS
(2023). https://dl.acm.org/doi/pdf/10.1145/3589763[26] Sijie Tao, Nuo Chen, Tetsuya Sakai, Zhumin Chu, Hiromi Arai, Ian Soboroff,
Nicola Ferro, and Maria Maistro. 2023. Overview of the NTCIR-17 FairWeb-1
Task. In Proceedings of NTCIR-17. to appear. https://doi.org/10.20736/0002001318[27] Julián Urbano, Harlley Lima, and Alan Hanjalic. 2019. A New Perspective on
Score Standardization. In Proceedings of ACM SIGIR 2019 (Paris, France). ACM,
New York, NY, USA, 1061–1064.
[28] William Webber, Alistair Moffat, and Justin Zobel. 2008. Score Standardization
for Inter-Collection Comparison of Retrieval Systems. In Proceedings of ACM
SIGIR 2008 (Singapore, Singapore). ACM, 51–58. https://doi.org/10.1145/1390334.
1390346[29] Shuo Zhang, Mu-Chun Wang, and Krisztian Balog. 2022. Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems.
In Proceedings of the 45th International ACM SIGIR Conference on Research andDevelopment in Information Retrieval (Madrid, Spain). Association for Computing
Machinery, 133–143.

[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom
Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson
Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse,
Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris
Olah, and Jared Kaplan. 2021. A General Language Assistant as a Laboratory for
Alignment. arXiv:2112.00861 [cs.CL][2] Leif Azzopardi, Diane Kelly, and Kathy Brennan. 2013. How Query Cost Affects
Search Behavior. In Proceedings of the 36th International ACM SIGIR Conference
on Research and Development in Information Retrieval (Dublin, Ireland) (SIGIR’13). ACM, 23–32.
[3] Leif Azzopardi and Guido Zuccon. 2018. Economic Models of Interaction. In
Computational Interaction, Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, andAndrew Howes (Eds.). Oxford University Press.
[4] Feza Baskaya, Heikki Keskustalo, and Kalervo Järvelin. 2012. Time Drives Interaction: Simulating Sessions in Diverse Searching Environments. In Proceedingsof the 35th International ACM SIGIR Conference on Research and Development inInformation Retrieval (Portland, Oregon, USA). ACM, 105–114.
[5] Christine Bauer, Ben Carterette, Nicola Ferro, and Norbert Fuhr. 2023. Reportfrom Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation
for Research and Education. arXiv:2305.01509 [cs.IR][6] Olivier Chapelle, Donald Metzler, Ya Zhang, and Pierre Grinspan. 2009. Expected
Reciprocal Rank for Graded Relevance. In Proceedings of ACM CIKM 2009. 621–
630.
[7] Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022.
Fairness in Information Access Systems. arXiv:2105.05779 [cs.IR][8] Michael D Ekstrand, Graham McDonald, and Amifa Raj. 2022. Overview of the
TREC 2021 Fair Ranking Track. In The Thirtieth Text REtrieval Conference (TREC
2021) Proceedings. NIST.
[9] Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini,
Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin
Potthast, Benno Stein, and Henning Wachsmuth. 2023. Perspectives on Large
Language Models for Relevance Judgment. In Proceedings of the 2023 ACM SIGIR
International Conference on Theory of Information Retrieval (Taipei, Taiwan). ACM,
39–50.
[10] David Griol, Javier Carbó, and José M. Molina. 2013. AN AUTOMATIC DIALOGSIMULATION TECHNIQUE TO DEVELOP AND EVALUATE INTERACTIVECONVERSATIONAL AGENTS. Applied Artificial Intelligence 27, 9 (2013), 759–
780.
[11] Ryuichiro Higashinaka, Kotato Funakoshi, Yuka Kobayashi, and Michimasa Inaba.
2016. The dialogue breakdown detection challenge: Task description, datasets,
and evaluation metrics. In Proceedings of the Tenth International Conference on
Language Resources and Evaluation (LREC’16) (Portorož, Slovenia). European
Language Resources Association (ELRA), 3146–3150.
[12] Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluationof IR Techniques. ACM Trans. Inf. Syst. 20, 4 (2002), 422–446. https://doi.org/10.
1145/582415.582418[13] Makoto P. Kato, Matthew Ekstrand-Abueg, Virgil Pavlu, Tetsuya
Sakai, Takehiro Yamamoto, and Mayu Iwata. 2014.
Overview of
the NTCIR-11 MobileClick Task. In Proceedings of NTCIR-11. 195–207.
https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/pdf/NTCIR/OVERVIEW/01-NTCIR11-OV-MOBILECLICK-KatoM.pdf[14] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar,
Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove,
Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson,
Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu

13