Why you should participate in the
NTCIR-18 FairWeb-2 Task
Version 20240326

FairWeb-2 organisers:Sijie Tao, Tetsuya Sakai, Junjie Wang, Hanpei Fang, Yuxiang Zhang(Waseda University),
Haitao Li, Yiteng Tu (Tsinghua University, China),
Nuo Chen (The Hong Kong Polytechnic University, China),
Maria Maistro (University of Copenhagen, Denmark)fairweb2org@list.waseda.jp
sakailab.com/fairweb2/

OUTLINE
1. Motivation
2. Web Search Subtask
3. Conversational Search Subtask
4. SummaryWeb search that’s “fair”“You are serving as a general chair of an IR conference. You
want to hire diverse IR researchers as organisers.”
Diversity dimensions:- Different career stage (include junior researchers, not justfamous researchers)- Different genders- Different countries
etc.

Screenshot taken on 14th July, 2022

h-index > 100

We only get famous and high h-index people…

h-index > 80

Screenshot taken on 14th July, 2022

Poor genderbalance…

Let’s consider group fairness
Attribute set: HINDEX

Attribute set: GENDER

Target distribution: give more
exposure to juniorresearchers!

Group 1
x < 10Group 210 ≦x < 30

Group 330 ≦x < 50

Approximates true gender: onlyreflects whether “he” or “she” isfound in the researcher bio

Group 450 ≦ xTarget distribution: equal
opportunities for different
genders!

Group 1

Group 2hesheGroup 3pronounnot foundWeb search that considers group fairness
query
SERP’s achieved
distribution

SERP(Search Engine Result Page)

Many relevant pages
near the top(traditional adhoc IR)

AND the achieved
distribution should be
similar to the target one

Target distribution

Handling ordinal groups properly
See Sakai21ACL, Sakai21CIKMlq
0.7

0.7
Achieved: D1

0.1

0.1

0.1

low

h-indexhigh

0.7
Target: D*

0.1

0.1

0.1

0.1

0.1

h-indexhighlowh-index0.1
low

Achieved: D2

If divergences for nominal groups (e.g. Jensen-Shannon Divergence) are used…
JSD = 0.3651

JSD = 0.3651

Divergences for ordinal groups can tell the difference
NMD = 0.2000
RNOD = 0.5477

Closer to target
Closer to target

NMD = 0.6000
RNOD = 0.6000

highSubtasks• Web Search (same as FairWeb-1)- Input: Researcher(R)/Movie(M)/YouTube(Y) topic- Output: SERP (Search Engine Result Page)
where the target corpus is Chuweb21D• Conversational Search (NEW)- Input: R/M/Y topic (same topic set as Web Search)- Output: User and system turns (English text, up to 1,250
words)

Conversational search that’s “fair”
1st user turn

System turns’ achieved
distribution

1st system turn2nd user turn2nd system turnTarget distribution

Topic types and attribute sets (same as
FairWeb-1 for continuity)
M (movie):

RATINGS (ordinal, 4 groups)
ORIGIN (nominal, 8 groups)#ratings on IMDbx < 100
100 ≦ x < 10K
10K ≦ x < 1M
1M ≦ xR (researcher): HINDEX (ordinal, 4 groups)
GENDER (nominal, 3 groups)
Y (YouTube):

SUBSCS (ordinal, 4 groups)

Countries of origin on
IMDb mapped to 8
geographic regions(one movie may cover
multiple countries)

e.g. a UK-Japan movie => Asia, Europe

Example topics from FairWeb-1
Researcher entities• R-topic"Researchers who have coauthored at least one paper with
Stephen E. Robertson, the inventor of BM25."
Movie entities• M-topic"sci-fi movies that feature time travel, listed in IMDb."
Video entities• Y-topic"Covers of songs written by Burt Bacharach and Hal David,
available on YouTube. Only Bacharach/David collaborations(just these two writers) are relevant."OUTLINE
1. Motivation
2. Web Search Subtask
3. Conversational Search Subtask
4. Summary

Subtask workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for each
topic)

SERP
SERP
SERP
SERP
n topics

SERP
SERP
SERP
SERP

SERP
SERP
SERP
SERP

Subtask workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for each
topic)
Form a depth-k pool foreach topic

n topics

pool
pool
pool
pool

SERP
SERP
SERP
SERP
n topics

SERP
SERP
SERP
SERP

SERP
SERP
SERP
SERP

Subtask workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for each
topic)
Form a depth-k pool foreach topic
Annotate up to 3 relevant
entities from eachdocument
Relevant entity

Web pageWeb page
Relevant entity

Subtask workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for each
topic)
Form a depth-k pool foreach topic
Annotate up to 3 relevant
entities from eachdocument

Relevant entity

Web page

Derive relevance + groupmembership of each page,compute evaluation
measures for each run

Relevant entity

Page relevanceGroup membership

Relevant entity schema(key attributes are underlined)
R(GoogleScholarURL, h-index,
ResearcherName, BioURL, he/she/other)
M(MovieName, IMDbURL, #ratings, countries)
Y(YouTubeContentURL, #subscribers)

Annotators will
use their favourite
search engines
to locate BioURLs,
GoogleSholarURLs,
and IMDbURLs.

FAIRE: FAIRE stands for Annotation Interface for Relevant Entities

Annotation interface (R topic)
prev

next

topic

Current doc
Document viewer

Document selectionpanel
researchername

GScholarURL

h-index

BioURLgenderresearchername

GScholarURL

h-index

BioURLgenderresearchername

GScholarURL

h-index

BioURLgenderassessor remarks
Backend records: <topicID, docID, ResearcherName, GScholarURL, h-index, BioURL, he/she/other>

Deriving page relevanceIf 2 annotators each find 3
nonoverlapping relevant entities

d: page
E(d): set of relevant entities extracted from d (|E(d)|≦6)r(e) ∈ {1,2}: relevance level of e ∈ E(d)Page relevance level g(d) ∈ {0,1,2} defined as follows
Max relevance level G=2Page relevance level = max entity relevance level within page

Deriving page group membershipGroup 1C = {C1 , … C|C| }: attribute set
F(e, C i): flag that maps e to exactly one group

Hard group membershipfor entities

A researcher whose h-index=5 (C=HINDEX):
F(e, C1)=1, F(e, C2) = F(e, C3) = F(e, C4) = 0
Ad researcher whose bio says ”he” (C=GENDER):
Uniform for nonrelevant pageF(e, C1)=1, F(e, C2) = F(e, C3) = 0

Group membership probabilities of d:C1 C2 C3 C4
e1
e2
e3
2/3

1/3

Deriving page group membership
For e, if Countries of Origin = {UK, Russia}
UK → Europe
Russia → Asia, Europe⇒ ORIGIN(e) = {Asia, Europe}

C = {C1 , … C|C| }: attribute set
ORIGIN(e) (⊆ C): set of geo regions
for movie e ∈ E(d) (m = |ORIGIN(e)| (≧1))
Soft group membership wrt ORIGIN for movie entities

Uniform for nonrelevant pageC1 C2 C3 C4
e1
e2
e3
1.5/3

0.5

0.5

1
0.5

0.5/3

0.5

1.0/3

GFR (Group Fairness and Relevance)
Sakai+23TOIS
L: SERPd L@k : doc at rank k in L

Page relevance level

Probability that users
will be satisfied withdoc at k

Probability that users will reach k
and finally get satisfied

Normalised Cumulative Utility (Sakai+08EVIA) e.g. ERR0, ¼, ¾ in our task

Similarity between achieved
distribution@k and target

Decay: probability that the users will abandon the
SERP at rank k
ERR (Expected Reciprocal Rank) user model
User
population
satisfied0

Decay L@1 = 0
1
Decay L@2 = 1 * (1/4) = 1/4
3/4

Decay L@3 = 1 * (3/4) * 0 = 0
1
Decay L@4 = 1 * (3/4) * 1 * (3/4)= 8/161/4

1/4
0

Nonrelevant(2^1 – 1)/2^2 = 1/4

Partially rel
Nonrelevant

3/4
Relevant(2^2 – 1)/2^3 = 3/4

Utility: how useful was the top k of the
SERP?
(Φ=0.99)

Nonrelevant

1/2 = 0.50

Partially rel

0.99^2 = 0.98

Nonrelevant

1/4 = 0.25

Relevant

0.99^4 = 0.96

1

0.5

1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
88
91
94
97
100

0

1/k

φ^k

Rank

DistrSim: Similarity between achieved distribution@k and target
For the m-th attribute set:

0.7

0.7

Achieved distribution
at rank k of SERP

0.1

0.1

0.1

Target distribution
0.1

0.1

Similarity: larger=better• For attribute sets containing nominal groups:
Divegence= JSD (Jensen-Shannon Divergence)• For attribute sets containing ordinal groups:
Divegence= NMD (Normalised Match Distance) or
RNOD (Root Normalised Order-aware Divergence)

0.1

JSD etc. are not suitable for ordinal groups
0.7

0.7
Achieved: D1

0.1

D1’

0.1

0.1

0.4

0.4

0.7
Target: D*
0.1

Achieved: D2
0.1

0.1

0.1

0.1

0.1

0.1

0.4

0.4
0.1

0.1

0.1

D2’

Log with base 2KLD(D1||D1’) =0.1log(0.1/0.4) + 0.7log(0.7/0.4) = 0.3651
KLD(D*||D1’) =0.7log(0.7/0.4) + 0.1log(0.1/0.4) = 0.3651
JSD = (0.3651+0.3651)/2 = 0.3651

KLD(D2||D2’) =0.1log(0.1/0.4) + 0.7log(0.7/0.4) = 0.3651
KLD(D*||D2’) =0.7log(0.7/0.4) + 0.1log(0.1/0.4) = 0.3651
JSD = (0.3651+0.3651)/2 = 0.3651

D1 (not too bad) and D2 (terrible) considered equivalentaka Earth Mover’s DistanceNMD (Normalised Match Distance)
0.7

0.7
Achieved: D1

0.1

0.1

(0.1,

0.8,

Target: D*

0.1

Cumulative:

0.7

0.1

0.1

Achieved: D2

0.1

1.0)

(0.7,

NMD =
( |0.1-0.7| + |0.8-0.8| +|0.9-0.9| + |1.0-1.0| )/3= 0.6/3
Closer to the target!
= 0.2000

0.1

0.1

0.2,

0.3,

Cumulative:

Cumulative:
0.9,

0.1

0.8,

0.9,

1.0)

(0.1,

1.0)

NMD =
( |0.1-0.7| + |0.2-0.8| +|0.3-0.9| + |1.0-1.0| )/3=(0.6+0.6+0.6)/3= 0.6000

For computing RNOD

DW (Distance-Weighted sum of squares)
0.7

0.7
Achieved: D1

0.1

0.1

0.1

j=3
j=4
DW1 = j=2
1*(0.7-0.1)^2 + 2*(0.1-0.1)^2 + 3*(0.1-0.1)^2= 0.36
j=3
j=4
DW2 = j=1
1*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 2*(0.1-0.1)^2= 0.36
j=2
j=4
DW3 = j=1
2*(0.1-0.7)^2 + 1*(0.7-0.1)^2+ 1*(0.1-0.1)^2= 1.08j=2
j=3
DW4 = j=1
3*(0.1-0.7)^2 + 2*(0.7-0.1)^2 + 1*(0.1-0.1)^2= 1.800.7
Target: D*
0.1

0.1

Achieved: D2

0.1

0.1

0.1

0.1

j=2
j=3
j=4
DW1 =
1*(0.1-0.1)^2 + 2*(0.1-0.1)^2 + 3*(0.7-0.1)^2= 1.08
j=1
j=3
j=4
DW2 =
1*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 2*(0.7-0.1)^2= 1.08
j=1
j=2
j=4
DW3 =
2*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 1*(0.7-0.1)^2= 1.08
j=1
j=2
j=3
DW4 =
3*(0.1-0.7)^2 + 2*(0.1-0.1)^2 + 1*(0.1-0.1)^2= 1.08RNOD (Root Normalised Order-aware Divergence)
0.7

0.7
Achieved: D1

0.1

0.1

0.1

DW (0.36, 0.36, 1.08, 1.08)

OD =(0.36+0.36+1.08+1.80)/4= 0.90RNOD =SQRT(0.90/3)= 0.5477

0.7
Target: D*
0.1

0.1

Achieved: D2

0.1

0.1

0.1

DW (1.08, 1.08, 1.08, 1.08)

OD =(1.08+1.08+1.08+1.08)/4=1.08
RNOD =SQRT(1.08/3)= 0.6000

Closer to the target!

0.1

Evaluating intersectional group fairness
GF(HINDEX)GF(RATINGS)• R topics
relevance
HINDEX (ordinal)
GENDER (nominal)• M topics
relevance
RATINGS (ordinal)
ORIGIN (nominal)

GF(GENDER)

GF(ORIGIN)

How is GFR different from the single-ranking
measure used at TREC 2022?
https://fair-trec.github.io/docs/Fair_Ranking_2022_Participant_Instructions.pdf

Main diffs:• Decay: TREC uses nDCG decay (relevance-unaware); we use ERR
decay (relevance-aware).
• Achieved distribution: TREC computes a single distribution for a
SERP; we obtain a distribution at every rank with a relevant doc.
• Divergence: TREC uses JSD; we use JSD for nominal groups but
NMD and RNOD for ordinal groups.
• Combining relevance and group fairness: TREC multiplies the two;
we average relevance (ERR or iRBU score) and one or two GF
scores.

OUTLINE
1. Motivation
2. Web Search Subtask
3. Conversational Search Subtask
4. SummaryOutput: user and system turns(R-topic: researchername + Google Scholar URL)
Example R-topic: “Researchers who have coauthored at least one paper with Stephen E.
Robertson, the inventor of BM25.”
Example output:

Must include both user turns (prompts) and system turns

U: Please return a list of researchers who have coauthored at least one paper with Stephen E.
Robertson. Each researcher name must be accompanied by a Google Scholar URL.
Format: <researcher name>, <Google Schlar URL>S:
Karen Sparck Jones, https://scholar.google.com/citations?user=HzDsaGYAAAAJ
Gareth Jones, https://scholar.google.com/citations?user=YJuN_H8AAAAJU: Please name more.
S:

Muti-turn (optional)

Each user turn must include “U:” as a prefix.
Each system turn must include “S:” as a prefix.
Tetsuya Sakai, https://scholar.google.com/citations?user=vi56gz4AAAAJ
Each output must begin with a “U:”
Up to 1,250 words (≒ 5 minutes reading time)#words will be counted based on white spaces
Output may be single-turn or multi-turnOutput: user and system turns(M-topic: movienename + IMDbURL)
Example M-topic: “sci-fi movies that feature time travel, listed in IMDb.”
Example output:
U: Please return a list of sci-fi movies that feature time travel. Each movie must e accompanied by an IMDb URL.
Format: <movietitle>,<IMDb URL>
S:
Here are some sci-fi movies that feature time travel, listed in IMDb:
Back to the Future (1985), https://www.imdb.com/title/tt0088763/
The Terminator (1984), https://www.imdb.com/title/tt0088247/
Groundhog Day (1993), https://www.imdb.com/title/tt0107048/
Interstellar (2014), https://www.imdb.com/title/tt0816692/

Each user turn must include “U:” as a prefix.
Each system turn must include “S:” as a prefix.
Each output must begin with a “U:”
Up to 1,250 words (≒ 5 minutes reading time)#words will be counted based on white spaces

The Time Machine (1960), https://www.imdb.com/title/tt0054387/
I hope this helps! Let me know if there’s anything else I can help you with.

Output: user and system turns(Y-topic: videotitle + youtubeURL)
Example Y-topic: “Covers of songs written by Burt Bacharach and Hal David, available
on YouTube. Only Bacharach/David collaborations (just these two writers) are relevant”
Example output:
U: Please return a list of covers of Bacharach/David songs. Each cover must be accompanied by a YouTube URL.
Format: <youtube title>, <youtube URL>
S: Here are the covers of songs written by Burt Bacharach and Hal David that are available on YouTube:
Walk On By, https://www.youtube.com/watch?v=AO073fekFJk
I Say a Little Prayer, https://www.youtube.com/watch?v=KtBbyglq37E
Raindrops Keep Fallin’ on My Head, https://www.youtube.com/watch?v=5DmYLrxR0Y8
What the World Needs Now Is Love, https://www.youtube.com/watch?v=YUaxVQPohlU
Alfie, https://www.youtube.com/watch?v=ZVMIk3xYaYo

U: Can you name a few more?
S:
Walk On By, https://www.youtube.com/watch?v=AO073fekFJk

Duplicate entities will not be rewarded

Each user turn must include “U:” as a prefix.
Each system turn must include “S:” as a prefix.
Each output must begin with a “U:”
Up to 1,250 words (≒ 5 minutes reading time)#words will be counted based on white spaces

Subtask workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (a textual
conversation for each topic)
Collect all submitted
conversations for each topic
Annotate all entities(with URLs) in eachconversation

Relevant entity

Derive relevance + groupmembership of eachconversation, computeevaluation measures foreach run

Conversation
Relevant entity

Relevance

Group membership

Annotation interface (R topic)
prev

Current conversation

Conversation
selection panel

next

topic

Convertation viewer

Annotator can highlight aregion and enter:Researchername
GoogleScholarURL,
h-index,
BiographyURL,
genderKaren Sparck Jones,
https://en.wikipedia.org/wiki/Karen_Sp%C3%A4rck_Jones

Position = p
if this URL is thep-th word
assessor remarks

Backend records: <topicID, docID, position, ResearcherName, GScholarURL, h-index, BioURL, he/she/other>Position (white-space-based)
User prompt is included in

1U:
the word count
2Please3return(to reflect user effort):
4a
Try to make it short
5list
6of
7covers
8of
9Bacharach/David
10songs.
11Each12cover
13must14be
15accompanied
16by
17a
18YouTube
19URL.
20Format:21<youtube title>,
Position of
22<youtube URL>
1st relevant
23S:

entity24Here25are26the27covers
28of
29songs
30written
31by
32Burt33Bacharach
34and
35Hal
36David37that38are
39available
40on
41YouTube:
42Walk
43On
44By,
45https://www.youtube.com/watch?v=AO073fekFJkFirst relevant
entityGFRC: GFR for Conversations [Sakai23EVIA]: T-round user-system conversation: j-th nugget (=relevant piece of text) in i-th system turn=relevant entity

Gain value of nugget
Relevance
score

Normalisation factor

Position-based weight of nugget(position defined wrt C)GFRC: R instantiation
Average reading speed (English): 250 words/minute
Given 5 minutes, up to L=1,250 words can be read(Nuggets after 1,250 words are worthless)Entities early in the conversation are rewarded more

1

When every word is a relevant
nugget (unattainable)

pw
0

1

WC (including all user-system turns)

L+1

GFRC: GFR for Conversations
Group Fairness
score

When there are M attribute sets to consider

Normalisation
factorPosition-based weight of turn(position defined wrt C)

Weight for them-th attributesetDistribution
similarity between
achieved andtarget (for the m-th
attribute set)

GFRC: GF instantiation

Simply average DistrSim’s overrelevant system turns

As in Web Search, use JSD for nominal and NMD/RNOD for ordinal groups

GFRC: examples from
Sakai23EVIA

Nuggets in blue
Annotations in redst
1 system turn
Bing

Here are some sci-fi movies that feature time travel, listed in IMDb:
Back to the Future (1985): https://www.imdb.com/title/tt0088763/#L2

Sure, here are some sci-fi movies that feature time travel, listed in IMDb, with their IMDB links
and posters:

wc=35

Back to the Future (1985): A nerdy teenager accidentally travels back in time to 1955 andmeets his future parents, jeopardizing his own existence.

#RATINGS: (0,0,0,1) #1.3M

Back to the Future (1985) movie posterOpens in a new window#ORIGIN: (0,1,0,0,0,0,0,0) #United States

www.imdb.com

The Terminator (1984): https://www.imdb.com/title/tt0088247/

wc=39

Back to the Future (1985) movie poster#RATINGS: (0,0,1,0) #898K

Pulp Fiction (1994): A series of interconnected stories in Los Angeles's underworld. One of thestories involves a mob hit that goes wrong and the hitman's attempt to fix it by traveling backin time.

#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States

Pulp Fiction (1994) movie posterOpens in a new window#L2

Groundhog Day (1993): https://www.imdb.com/title/tt0107048/

wc=43

Googlewww.originalfilmart.com
Pulp Fiction (1994) movie poster#L2:#RATINGS: (0,0,1,0) #661K#ORIGIN: (0,1,0,0,0,0,0,0) #United States
Interstellar (2014): https://www.imdb.com/title/tt0816692/

#L1

wc=46

#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
The Time Machine (1960): https://www.imdb.com/title/tt0054387/#L2#RATINGS: (0,0,1,0) #43K
#ORIGIN: (0,1,0,0,0,0,0,0) #United States
I hope this helps! Let me know if there’s anything else I can help you with.

wc=51

A 452-word response that does not follow
the specified format

Nuggets in blue
Annotations in redst
1 system turn
Bing

Here are some sci-fi movies that feature time travel, listed in IMDb:
Back to the Future (1985): https://www.imdb.com/title/tt0088763/

Highly relevant#L2Sure, here are some sci-fi movies that feature time travel, listed in IMDb, with their IMDB links
and posters:

wc=35

Back to the Future (1985): A nerdy teenager accidentally travels back in time to 1955 andmeets his future parents, jeopardizing his own existence.

#RATINGS: (0,0,0,1) #1.3M

Back to the Future (1985) movie posterOpens in a new window#ORIGIN: (0,1,0,0,0,0,0,0) #United States

www.imdb.com

The Terminator (1984): https://www.imdb.com/title/tt0088247/

wc=39

Back to the Future (1985) movie poster#RATINGS: (0,0,1,0) #898K

Pulp Fiction (1994): A series of interconnected stories in Los Angeles's underworld. One of thestories involves a mob hit that goes wrong and the hitman's attempt to fix it by traveling backin time.

#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States

Pulp Fiction (1994) movie posterOpens in a new window

Groundhog Day (1993): https://www.imdb.com/title/tt0107048/

www.originalfilmart.com#L2

Pulp Fiction (1994) movie poster#L2

GoogleThe Terminator (1984): https://www.imdb.com/title/tt0088247/
wc=43
#L2
#RATINGS: (0,0,1,0) #898K#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States:#RATINGS: (0,0,1,0) #661K#ORIGIN: (0,1,0,0,0,0,0,0) #United States

Interstellar (2014): https://www.imdb.com/title/tt0816692/

#L1

wc=46

#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
The Time Machine (1960): https://www.imdb.com/title/tt0054387/#L2#RATINGS: (0,0,1,0) #43K
#ORIGIN: (0,1,0,0,0,0,0,0) #United States
I hope this helps! Let me know if there’s anything else I can help you with.

wc=51

A 452-word response that does not follow
the specified format

st
1 system turn
Bing

Here are some sci-fi movies that feature time travel, listed in IMDb:
Back to the Future (1985): https://www.imdb.com/title/tt0088763/#L2

Nuggets in blue
Annotations in red
Group 1:
x < 100
Group 2:100 ≦ x < 10K
GoogleGroup 3:
10K ≦ x < 1M
Group 4:
1M ≦ x898K ratings on IMDb ⇒ Group 3Sure, here are some sci-fi movies that feature time travel, listed in IMDb, with their IMDB links
and posters:

wc=35

Back to the Future (1985): A nerdy teenager accidentally travels back in time to 1955 andmeets his future parents, jeopardizing his own existence.

#RATINGS: (0,0,0,1) #1.3M

Back to the Future (1985) movie posterOpens in a new window#ORIGIN: (0,1,0,0,0,0,0,0) #United States

www.imdb.com

The Terminator (1984): https://www.imdb.com/title/tt0088247/

wc=39

Back to the Future (1985) movie poster#RATINGS: (0,0,1,0) #898K

Pulp Fiction (1994): A series of interconnected stories in Los Angeles's underworld. One of thestories involves a mob hit that goes wrong and the hitman's attempt to fix it by traveling backin time.

#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States

Pulp Fiction (1994) movie posterOpens in a new window

Groundhog Day (1993): https://www.imdb.com/title/tt0107048/

www.originalfilmart.com#L2

Pulp Fiction (1994) movie poster#L2

The Terminator (1984): https://www.imdb.com/title/tt0088247/
wc=43
#L2
#RATINGS: (0,0,1,0) #898K#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States:#RATINGS: (0,0,1,0) #661K#ORIGIN: (0,1,0,0,0,0,0,0) #United States

Interstellar (2014): https://www.imdb.com/title/tt0816692/

#L1

wc=46

#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
The Time Machine (1960): https://www.imdb.com/title/tt0054387/#L2#RATINGS: (0,0,1,0) #43K
#ORIGIN: (0,1,0,0,0,0,0,0) #United States
I hope this helps! Let me know if there’s anything else I can help you with.

wc=51

A 452-word response that does not follow
the specified format

Group 1: Africa
Nuggets in blue
Group 2: America
Annotations in red
Group 3: Antarctica
GoogleGroup 4: Asia
wc=35
Group 5: CaribbeanGroup 6: Europe
Group 7: Middle East
wc=39
Group 8: Oceania
The Terminator (1984): https://www.imdb.com/title/tt0088247/
Countries of origin on IMDB:
wc=43
US + UK ⇒ Groups 2 and 6#L2st
1 system turn
Bing

Here are some sci-fi movies that feature time travel, listed in IMDb:

Sure, here are some sci-fi movies that feature time travel, listed in IMDb, with their IMDB links
and posters:

Back to the Future (1985): https://www.imdb.com/title/tt0088763/#L2

Back to the Future (1985): A nerdy teenager accidentally travels back in time to 1955 andmeets his future parents, jeopardizing his own existence.

#RATINGS: (0,0,0,1) #1.3M

Back to the Future (1985) movie posterOpens in a new window#ORIGIN: (0,1,0,0,0,0,0,0) #United States

www.imdb.com

The Terminator (1984): https://www.imdb.com/title/tt0088247/

Back to the Future (1985) movie poster#L2#RATINGS: (0,0,1,0) #898K

Pulp Fiction (1994): A series of interconnected stories in Los Angeles's underworld. One of thestories involves a mob hit that goes wrong and the hitman's attempt to fix it by traveling backin time.

#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United States

Pulp Fiction (1994) movie posterOpens in a new window

Groundhog Day (1993): https://www.imdb.com/title/tt0107048/

www.originalfilmart.com#L2

Pulp Fiction (1994) movie poster#RATINGS: (0,0,1,0) #898K#ORIGIN: (0,1/2,0,0,0,1/2,0,0) #United Kingdom,United State:#RATINGS: (0,0,1,0) #661K#ORIGIN: (0,1,0,0,0,0,0,0) #United States

Interstellar (2014): https://www.imdb.com/title/tt0816692/

#L1

wc=46

#RATINGS: (0,0,0,1) #2M#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada
The Time Machine (1960): https://www.imdb.com/title/tt0054387/#L2#RATINGS: (0,0,1,0) #43K
#ORIGIN: (0,1,0,0,0,0,0,0) #United States
I hope this helps! Let me know if there’s anything else I can help you with.

wc=51

A 452-word response that does not follow
the specified format

2nd system turnU2: Adhere to the output format I specified:"Format: movie title, iMDB URL"

U2: Can you name a few more?
Bing

Sure! Here are some more sci-fi movies that feature time travel, listed in IMDb:

Sure, here are the sci-fi movies that feature time travel, listed in IMDb, in the specified format:

12 Monkeys (1995): https://www.imdb.com/title/tt0114746/

Movie title

#L2

wc=91

Back to the Future (1985)#L2#ORIGIN: (0,1,0,0,0,0,0,0) #United States#RATINGS: (0,0,0,1) #1.3M

#L2

wc=96

https://www.imdb.com/title/tt0088763/

wc=506#RATINGS: (0,0,1,0) #634K

The Butterfly Effect (2004): https://www.imdb.com/title/tt0289879/#ORIGIN: (0,1,0,0,0,0,0,0) #United States
Back to the Future (1985) movie posterOpens in a new window#RATINGS: (0,0,1,0) #508K

www.imdb.com#ORIGIN: (0,1,0,0,0,0,0,0) #United States,Canada

Back to the Future (1985) movie poster

Looper (2012): https://www.imdb.com/title/tt1276104/

wc=99

Pulp Fiction (1994)

https://www.imdb.com/title/tt0110912/

#L2

#L0

#RATINGS: (0,0,1,0) #592K

Pulp Fiction (1994) movie posterOpens in a new window#ORIGIN: (0,1/2,0,1/2,0,0,0,0) #United States,China

www.originalfilmart.com

Edge of Tomorrow (2014): https://www.imdb.com/title/tt1631867/

Pulp Fiction (1994) movie poster#L2

wc=104

Looper (2012)https://www.imdb.com/title/tt1853739/#RATINGS: (0,0,1,0) #711K#L0 # wrong URL "You're Next"#ORIGIN: (0,1,0,0,0,0,0,0) #United States,Canada

Looper (2012) movie posterOpens in a new window

Predestination (2014): https://www.imdb.com/title/tt2397535/#L2

wc=107

GoogleIMDb URL

Wrong URL

www.imdb.com
Looper (2012) movie poster#RATINGS: (0,0,1,0) #294K

Interstellar (2014)#ORIGIN: (0,1/2,0,0,0,0,0,1/2)#Australia,United States#L1

I hope this helps! Let me know if there’s anything else I can help you with.

#RATINGS: (0,0,0,1) #2M

https://www.imdb.com/title/tt0816692/#ORIGIN: (0,2/3,0,0,0,1/3,0,0) #United States,United Kingdom,Canada:

wc=560R scores
Google

Bing

pw(n)

g(n)

pw(n)*g(n)No relevant entities in 1st turn

pw(n)

g(n)

pw(n)*g(n)Not really time travel

Only about 9.8% of Bing’s Rscore

GF scores - RATINGS (Bing vs Google)
Membership vectors of
5 movies:
0
0
0
1
0
0
1
0
0
0
1
0
0
0
0
1
0
0
1
0

Achieved distribution

Gold distribution: uniform
Membership vectors of 2 movies:
0
0
0
1
0
0
0
1

DistrSim(RNOD)

Achieved distribution

DistrSim(RNOD)

Google returns famous
movies only

GF scores - ORIGIN (Bing vs Google)
0.2Gold
distribution 0.1(based on0#countries)

Oceania

Middle East

EuropeCaribbean

Asia

Antarctica

America

AfricaOceania

Middle East

EuropeCaribbean

Asia

Antarctica

America

AfricaS1
S2

Google returns only American and European movies so the GF score is low.
Bing does better (also returns Asian and Oceanian movies).

OUTLINE
1. Motivation
2. Web Search Subtask
3. Conversational Search Subtask
4. Summary

Timeline (tentative)
Nov 1, 2024: test topics released; task registrations due
Dec 15, 2024: run submissions due
Dec 2024-Jan 2025: Entity annotation + evaluation
Feb 1, 2025: Evaluation results + draft overview released
June 10-13, 2025: NTCIR-18@NII, Tokyo

We know you care about both fairness and IR.
That’s why we know you will participate in this task!

Links• NTCIR-17 FairWeb-1 page (with official results, relevant
papers, etc.)• NTCIR-18 FairWeb-2 page (with relevant papers, etc.)• Lots of videos and slides on divergences for quantification
tasks, i.e., for comparing two probability mass functions(from ACL 2021)