Introducing
The NTCIR-17 FairWeb-1
Task
Tao Sijie, Nuo Chen, Tetsuya Sakai (Waseda University, Japan)
Zhumin Chu (Tsinghua University, P.R.C.)
Nicola Ferro (University of Padua, Italy)
Maria Maistro (University of Copenhagen, Denmark)Ian Soboroff (NIST, USA)
Hiromi Arai (RIKEN AIP, Japan)
fairweb1org@list.waseda.jp
http://sakailab.com/fairweb1/

Version 20220925

TOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. Summary

“You are serving as a general chair of an IR conference. You
want to hire diverse IR researchers as organisers.”
Diversity dimensions:- Different career stage (include junior researchers, not justfamous researchers)- Different genders- Different countries
etc.

Screenshot taken on 14th July, 2022

h-index > 100

We only get famous and high h-index people…

h-index > 80

Screenshot taken on 14th July, 2022

Poor genderbalance…

Let’s consider group fairness
Attribute set: HINDEX

Attribute set: GENDER

Target distribution: give more
exposure to juniorresearchers!

Group 1x < 10Group 2
10 ≦x < 30Group 3
30 ≦x < 50Group 4
50 ≦ xTarget distribution: equal
opportunities for different
genders!

Group 1

Group 2

hesheGroup 3otherWeb search that considers group fairness
query
SERP’s achieved
distribution

SERP(Search Engine Result Page)

Many relevant pages
near the top(traditional adhoc IR)

AND the achieved
distribution should be
similar to the target one

Target distribution

Handling ordinal groups properly
0.7

0.7
Achieved: D1

0.1

0.1

0.1

lowh-index

high0.7
Target: D*0.10.1

0.1

0.1

0.1

h-indexhighlowh-index0.1
low

Achieved: D2

If divergences for nominal groups (e.g. Jensen-Shannon Divergence) is used…
JSD = 0.3651

JSD = 0.3651

Divergences for ordinal groups can tell the difference
NMD = 0.2000
RNOD = 0.5477

Closer to target
Closer to target

NMD = 0.6000
RNOD = 0.6000

highTOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. SummaryInput/Output
INPUT- Search topic: describes an information need about entities
that satisfy a certain condition- One or more attribute set and a target distribution(probability mass function) for each of them
OUTPUT- Run: a SERP for each topic (TREC format). Expected tocontain relevant documents near top ranks AND to be groupfair wrt each attribute set

Task workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for eachtopic)

SERP
SERP
SERP
SERP
n topics

SERP
SERP
SERP
SERP

SERP
SERPSERP
SERP

Task workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for eachtopic)
Form a depth-k pool foreach topic

n topics

poolpoolpoolpool

SERP
SERP
SERP
SERP
n topics

SERP
SERP
SERP
SERP

SERP
SERP
SERP
SERP

Task workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for eachtopic)
Form a depth-k pool foreach topic
Annotate up to 3 relevant
entities from eachdocument
Relevant entity

Web pageWeb page
Relevant entity

Task workflow
Annotators

Participants

Organisers
Release the topic set

Submit runs (SERP for eachtopic)
Form a depth-k pool foreach topic
Annotate up to 3 relevant
entities from eachdocument

Relevant entity

Web page

Derive relevance + groupmembership of each page,compute evaluation
measures for each run

Relevant entity

Page relevanceGroup membership

TOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. SummaryEntities“something that exists apart from other things, having its own
independent existence”
https://dictionary.cambridge.org/dictionary/english/entity
Four entity types considered at FairWeb-1:
R: researchers
M: movies
T: Twitter accounts
Y: YouTube contents

Relevant entities and topics
Relevant entity: entities that satisfy the condition specified in
the topic description.
Four topic types with examples:
R: information retrieval researchers
M: Daniel Craig 007 movies
T: Twitter accounts that provide info on COVID
Y: Coldplay covers on YouTube
The underlined parts indicate the topic type

Attribute sets for each topic type
• R topic: HINDEX (ordinal, 4 groups)
GENDER (nominal, 3 groups)• M topic: REVIEWS (ordinal, 4 groups)
ORIGIN (nominal, 8 groups)• T topic: FOLLOWERS (ordinal, 4 groups)• Y topic: SUBSCS (ordinal, 4 groups)

Attribute sets for each topic type
• R topic: HINDEX (ordinal, 4 groups)
GENDER (nominal, 3 groups)• M topic: REVIEWS (ordinal, 4 groups)
ORIGIN (nominal, 8 groups)• T topic: FOLLOWERS (ordinal, 4 groups)
Googlescholar SUBSCSh-index• Y topic:(ordinal, 4 groups)
Gender label (he/she/other)x < 1010 ≦ x < 3030 ≦ x < 5050 ≦ x

based solely on what pronoun isused in the official researcher
bio
Note that this is just an approximationand simplification of true genderAttribute sets for each topic type
• R topic: HINDEX (ordinal, 4 groups)
AfricaGENDER (nominal, 3 groups)
America
Antarctica• M topic: REVIEWS (ordinal, 4 groups)
Asia
Caribbean
ORIGIN (nominal, 8 groups)
EuropeMiddle EastOceania• T topic: FOLLOWERS (ordinal, 4 groups)#reviews
on IMDbpage• Y topic:
SUBSCS(ordinal, 4 groups)
Countries of origin on IMDbx < 100100 ≦ x < 10K
10K ≦ x < 1M
1M ≦ xpage mapped to 8 geographic
regions (one movie may covermultiple countries)
e.g. a UK-Japan movie => Asia, Europe

Attribute sets for each topic type
• R topic: HINDEX (ordinal, 4 groups)#followers of twitter account
GENDER (nominal, 3 groups)(same grouping as REVIEWS)• M topic: REVIEWS (ordinal, 4 groups)
ORIGIN (nominal, 8 groups)• T topic: FOLLOWERS (ordinal, 4 groups)• Y topic: SUBSCS (ordinal, 4 groups)#subscribers of the creator(same grouping as REVIEWS)

TOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. Summary

Relevant entity schema
R(ResearcherName, BioURL, he/she/other,GoogleScholarURL, h-index)
M(MovieName, IMDbURL, #reviews, countries)
T(TwitterAccountURL, #followers)
Y(YouTubeContentURL, #subscribers)

Annotators will
use their favourite
search engines
to locate BioURLs,
GSholarURLs,andIMDbURLs.

Annotation interface (R topic)
prevnexttopic

Current doc
Document viewer

Document selection
panel

R entity key (if a researcher has multiple biopages, treat them as multiple researchers)

0 if no Gscholar page isfoundresearchername

BioURLgenderGScholarURL (or NA)

h-indexresearchername

BioURLgenderGScholarURL (or NA)

h-indexresearchername

BioURLgenderGScholarURL (or NA)

h-indexassessor remarks
Backend records: <topicID, docID, ResearcherName, BioURL, he/she/other, GScholarURL, h-index>

TOC1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. Summary

Deriving page relevanceIf 2 annotators each find 3
nonoverlapping relevant entities

d: page
E(d): set of relevant entities extracted from d (|E(d)|≦6)r(e) ∈ {1,2}: relevance level of e ∈ E(d)Page relevance level g(d) ∈ {0,1,2} defined as follows
Max relevance level G=2Page relevance level = max entity relevance level within page

Deriving page group membershipGroup 1C = {C1 , … C|C| }: attribute set
F(e, C i): flag that maps e to exactly one group

Hard group membershipfor entities

A researcher whose h-index=5 (C=HINDEX):
F(e, C1)=1, F(e, C2) = F(e, C3) = F(e, C4) = 0
Ad researcher whose bio says ”he” (C=GENDER):
Uniform for nonrelevant pageF(e, C1)=1, F(e, C2) = F(e, C3) = 0

Group membership probabilities of d:C1 C2 C3 C4e1e2e3
2/3

1/3

Deriving page group membership
C = {C1 , … C|C| }: attribute set (geo regions)
ORIGIN(e) (⊆ C): set of geo regions for movie e ∈ E(d)(m = |ORIGIN(e)| (≧1))
Soft group membership for movie entities
Uniform for nonrelevant pageC1 C2 C3 C4
e1
e2
e3
1.5/3

0.5

0.5

1
0.5

0.5/3

0.5

1.0/3

TOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. Summary

GFR (Group Fairness and Relevance)
https://arxiv.org/pdf/2204.00280.pdf
L: SERPd L@k : doc at rank k in L

Page relevance level

Probability that users
will be satisfied withdoc at k

Probability that users will reach k
and finally get satisfied

Relevance measures like ERR0, ¼, ¾ in our task

Similarity between achieved
distribution@k and target

Decay: probability that the users will abandon theSERP at rank k
ERR (Expected Reciprocal Rank) user model
User
population
satisfied
0Decay L@1 = 0
1
Decay L@2 = 1 * (1/4) = 1/4
3/4

Decay L@3 = 1 * (3/4) * 0 = 01
Decay L@4 = 1 * (3/4) * 1 * (3/4)= 8/16
1/4

1/4
0Nonrelevant(2^1 – 1)/2^2 = 1/4

Partially rel
Nonrelevant

3/4
Relevant(2^2 – 1)/2^3 = 3/4

Utility: how useful was the top k of theSERP?
(Φ=0.99)

Nonrelevant

1/2 = 0.50

Partially rel

0.99^2 = 0.98

Nonrelevant

1/4 = 0.25

Relevant

0.99^4 = 0.96

10.5147101316192225
2831343740434649525558616467707376
7982858891
94
97
10001/kφ^kRank

DistrSim: Similarity between achieved distribution@k and target
For the m-th attribute set:

0.7

0.7

Achieved distribution
at rank k of SERP

0.1

0.1

0.1

Target distribution
0.1

0.1

0.1

Similarity: larger=better• For attribute sets containing nominal groups:
Divegence= JSD (Jensen-Shannon Divergence)• For attribute sets containing ordinal groups:
Divegence= NMD (Normalised Match Distance) or
RNOD (Root Normalised Order-aware Divergence)
See Sakai’s CIKM2021LQ workshop paper: http://ceur-ws.org/Vol-3052/paper21.pdf

JSD etc. are not suitable for ordinal groups
0.7

0.7
Achieved: D1

0.1

D1’

0.1

0.1

0.4

0.4

0.7
Target: D*
0.1

Achieved: D2
0.1

0.1

0.1

0.1

0.1

0.1

0.4

0.4
0.1

0.1

0.1

D2’

Log with base 2KLD(D1||D1’) =0.1log(0.1/0.4) + 0.7log(0.7/0.4) = 0.3651
KLD(D*||D1’) =0.7log(0.7/0.4) + 0.1log(0.1/0.4) = 0.3651
JSD = (0.3651+0.3651)/2 = 0.3651KLD(D2||D2’) =0.1log(0.1/0.4) + 0.7log(0.7/0.4) = 0.3651
KLD(D*||D2’) =0.7log(0.7/0.4) + 0.1log(0.1/0.4) = 0.3651
JSD = (0.3651+0.3651)/2 = 0.3651

D1 (not too bad) and D2 (terrible) considered equivalentaka Earth Mover’s DistanceNMD (Normalised Match Distance)
0.7

0.7
Achieved: D1

0.1

0.1

(0.1,0.8,Target: D*

0.1

Cumulative:

0.7

0.1

0.1

Achieved: D2

0.11.0)(0.7,NMD =( |0.1-0.7| + |0.8-0.8| +|0.9-0.9| + |1.0-1.0| )/3= 0.6/3
Closer to the target!
= 0.2000

0.1

0.1

0.2,0.3,

Cumulative:

Cumulative:
0.9,0.1

0.8,0.9,1.0)(0.1,1.0)

NMD =( |0.1-0.7| + |0.2-0.8| +|0.3-0.9| + |1.0-1.0| )/3=(0.6+0.6+0.6)/3= 0.6000

For computing RNOD

DW (Distance-Weighted sum of squares)
0.7

0.7
Achieved: D1

0.1

0.1

0.1

j=3
j=4
DW1 = j=2
1*(0.7-0.1)^2 + 2*(0.1-0.1)^2 + 3*(0.1-0.1)^2= 0.36
j=3
j=4
DW2 = j=1
1*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 2*(0.1-0.1)^2= 0.36
j=2
j=4
DW3 = j=1
2*(0.1-0.7)^2 + 1*(0.7-0.1)^2+ 1*(0.1-0.1)^2= 1.08j=2
j=3
DW4 = j=1
3*(0.1-0.7)^2 + 2*(0.7-0.1)^2 + 1*(0.1-0.1)^2= 1.800.7
Target: D*
0.1

0.1

Achieved: D2

0.1

0.1

0.1

0.1

j=2
j=3
j=4
DW1 =1*(0.1-0.1)^2 + 2*(0.1-0.1)^2 + 3*(0.7-0.1)^2= 1.08j=1
j=3
j=4
DW2 =1*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 2*(0.7-0.1)^2= 1.08
j=1
j=2
j=4
DW3 =
2*(0.1-0.7)^2 + 1*(0.1-0.1)^2 + 1*(0.7-0.1)^2= 1.08
j=1
j=2
j=3
DW4 =
3*(0.1-0.7)^2 + 2*(0.1-0.1)^2 + 1*(0.1-0.1)^2= 1.08RNOD (Root Normalised Order-aware Divergence)
0.7

0.7
Achieved: D1

0.1

0.1

0.1

DW (0.36, 0.36, 1.08, 1.08)

OD =(0.36+0.36+1.08+1.80)/4= 0.90RNOD =SQRT(0.90/3)= 0.5477

0.7
Target: D*
0.1

0.1

Achieved: D2

0.1

0.1

0.1

DW (1.08, 1.08, 1.08, 1.08)

OD =(1.08+1.08+1.08+1.08)/4=1.08
RNOD =SQRT(1.08/3)= 0.6000

Closer to the target!

0.1

Evaluating intersectional group fairness
GF(HINDEX)GF(REVIEWS)• R topics
relevance
HINDEX (ordinal)
GENDER (nominal)• M topics
relevance
REVIEWS (ordinal)
ORIGIN (nominal)

GF(GENDER)

GF(ORIGIN)

How is GFR different from the single-ranking
measure used at TREC 2022?
https://fair-trec.github.io/docs/Fair_Ranking_2022_Participant_Instructions.pdf

Main diffs:• Decay: TREC uses nDCG decay (relevance-unaware); we use
ERR decay (relevance-aware)• Divergence: TREC uses JSD; we use JSD for nominal groups
but NMD and RNOD for ordinal groups• Combining relevance and group fairness: TREC multiplies thetwo; we average relevance (ERR or iRBU score) and one ortwo GF scores (and also look at the relationship acrosscomponents, e.g. ERR vs GF(HINDEX) vs GF(GENDER) ).

TOC
1. Motivation
2. Task overview
3. Entities, topics, attribute sets
4. Annotating relevant entities
5. Deriving page relevance and page group membership
6. Evaluation measures
7. Summary

Summary• Participants are expected to returns SERPs that arerelevant and group-fair!
• The FairWeb-1 task treats ordinal groups as ordinal,and considers intersectional group fairness between
ordinal groups and nominal groups.
• Ensuring fairness is researchers’ one big
responsibility! Please participate!

Timeline (tentative)
October 7, 2022:

Release of 1st CFP with sample topics and evaluation protocol

December 16, 2022:

Pilot relevance assessments for the sample topics and a few
pilot runs released; topic set size determined

Dec 19-Feb 28, 2023: Topic development
March 1, 2023:

Topics released; task registrations due

May 12, 2023:

Run submissions due

May 15-July 31, 2023: Entity annotations; runs evaluatedAugust 1, 2023:

Evaluation results and draft overview released

September 1, 2023:

Draft participant papers due

November 1, 2023:

Camera ready papers dueDecember 2023:

NTCIR-17@NII, Tokyo, JapanOn the history of NTCIR (open access book, 2020)
https://link.springer.com/content/pdf/10.1007/978-981-15-5554-1.pdf