RCIR @ NTCIR-16
Reading Comprehension in Information
Retrieval
Graham Healy, Dublin City University, Ireland
Cathal Gurrin, Dublin City University, IrelandNguyễn Thanh Binh, Ho Chi Minh University of Science, Vietnam+Tu-Khiem Le, Dublin City University, Ireland
Mai Boi Quach, Dublin City University, Ireland

Reading Comprehension
Can we tell from a person’s eye movements whether
they’ve comprehended the text they’ve read on
screen?

And can we use this information aspart of the retrieval process for text
content?

Eye movements when readingtext
When tracking a person’s eye movements when they
are reading, we observe different types of reading
behaviors …
Sequential reading of text
Scanning (with an information need)
Skimming (under pressure)…
Core text reading behaviours in the real worldEye movements when readingtext
And different types of reading behaviors areassociated with different patterns of comprehension

Reading Comprehension
Text Comprehension is multifaceted and complex: literal,inferential, predictive, evaluative, applied, …
We will focus on literal for this pilot task (more later).
So, by comprehension we mean whether they cananswer some questions about the text they’ve just read...

Day, R. R., & Park, J. S. (2005). Developing Reading Comprehension Questions. Reading ina foreign language, 17(1), 60-73.

RCIR @ NTCIR-16
Two sub-tasks
Sub-task 1:
Comprehensionevaluation task (CET)

Sub-task 2:
Comprehension-based
retrieval task (CRT)

Focus: Sort texts by
their comprehension
scores using biosignal
measures

Focus: Retrieve/rank
texts (for a variety oftopics) bycomprehension . score

Data Collection Experiment• 96 texts per participant X 10 participants• 4 induced reading conditions: sequential,scanning, skimming, proof readingExperiment/ Data
• 24 text topics (sourced from Wikipedia)• Each user-text pair with a measured
comprehension score• Mix of short and long texts• Training set of 480 trials (containing 12 topics)• Testing set of 480 trials (containing 12 topics and 12topics not in training set)Comprehension-evaluation Task
Sub-task 1:
Predict the comprehension score for each user-text pair in
the test set. Evaluated via e.g. MSE

Sub-task 2:
Retrieve topics (24) and rank by comprehension score.
Evaluated via e.g. Discounted Cumulative Gain

Data / Resources
Resources Provided:• Preprocessed (and raw) eye tracker data for eachtrial, pre-computed features, etc• Baseline system, code
• Submission system examples• Training / test
• And moreMeasures• Eye tracker, EOG, etc

Timeline*• Sept 2021 - Task Registration• Sept 2021 – Dataset Release• Dec 2021 – Test/dry runs
• Jan 2022 – Formal runs
• Feb 2022 - Evaluation result release• Feb 2022 – Draft Task Overview Paper release• Mar 2022 – Draft Task Participant Papers due• Apr 2022 – Camera ready submission due
This is a tentative timeline. The finalised timeline will be published at:ntcir-rcir.computing.dcu.ie

Thank you
Thank you for your time

• If you have any queries (or suggestions) please feel free to email me:
graham.healy@dcu.ie• Please check out ntcir-rcir.computing.dcu.ie (more detail soon)
Graham Healy, Dublin City University, Ireland
Cathal Gurrin, Dublin City University, IrelandNguyễn Thanh Binh, Ho Chi Minh University of Science, Vietnam+Tu-Khiem Le, Dublin City University, Ireland
Mai Boi Quach, Dublin City University, Ireland