NTCIR-16

Pilot Task: Session Search (SS)
††

†

‡

Yiqun Liu* , Jia Chen , Fan Zhang , Jiaxin Mao , Weizhi Ma†† Department of Computer Science & Tech., Tsinghua University‡ Gaoling School of AI, Renmin University of China
yiqunliu@tsinghua.edu.cn, chenjia0831@gmail.com

1Motivation• Multi-query sessions/tasks are common in search process nowadays.
• Considering the contextual information within sessions has been proved efficient for user intent
modeling in IR communities.
• Existing tasks in NTCIR have not involved session-based retrieval yet.

• Existing relevant tasks• TREC Session Track 2011-2014: small-scale datasets, simulated search tasks, only evaluate thelast query;• TREC Dynamic Domain (DD) Track 2015-2017: user simulators, ignore user reformulations.

==> NCTIR-16 Session Search Task

Methodology• To better assess the search effectiveness at both query-level and session-level, we aim to set up twosubtasks as follows:
Fully Observed Session Search (FOSS):Ø For a k-length session, we provide full session contexts in the first (k − 1) queries.
Ø NDCG, AP, RBP
Partially Observed Session Search (POSS):Ø In this subtask, we truncate all sessions before the last query. For a session with k queries (k ≥ 2),
we only reserve the session contexts in the first n queries, where 1 ≤ n ≤ k − 1.
Ø RS-DCG, RS-RBP

lsession. This setting follows TREC Session Tracks to enable ad-hoc evaluation by using metrics suchas NDCG, AP, and RBP, etc.
Partially Observed Session Search (POSS): In this subtask, we truncate all sessions before the lastquery. For a session with k queries (k ≥ 2), we only reserve the session contexts in the first n queries,
where 1 ≤ n ≤ k − 1. The value of n varies in different sessions. Participants will need to re-rank
documents for the last k − n queries(query) according to the partially observed contextual
information in previous search rounds. Session-level metrics such as RS-DCG and RS-RBP [3] will beadopted for the evaluation of system effectiveness.

Methodology

The differences between our Session Search task and previous related tasks are listed in the followingtable.
Table1. Differences between our Session Search task and previous related tasks
TREC Session Tracks
Ours NTCIR SS task
Number of Sessions• Chinese: 147,155 (Training) + 1,124 • English: 76~1,257(Validating) + 2,356 (Testing)Session Datasets• TianGong-ST• Session Track 2011-2014• TianGong-SS-FSD• An un-released field study dataset
Document Collection• A collection provided by TianGong- • ClueWeb09/ClueWeb12
ST, with more than 297,597 webpages.
• A collection which contains top 50
candidate documents for all queries
in the validating and testing set.
Source/Generationof • Refined from a search log from • Generated by real searchsession data
Sogou.com
users based on manuallydesigned topics.
• Extracted from two large-scale fieldstudies.
Support from log analysis Yes
Nofor annotation?
Support for session-level Yes
Noevaluation?

Expected Results:

Dataset & Resources• Training set:
• TianGong-ST (http://www.thuir.cn/tiangong-st/), Chinese-centric, with 147,155 refined Web search sessions, 40,596
unique queries, 297,597 Web pages, and six kinds of weak relevance labels assessed by click models.
• Participants can adopt any training techniques, e.g., adversarial learning, multi-task learning.

• Validating set:
• TianGong-SS-FSD (http://www.thuir.cn/tiangong-ss-fsd/), A Chinese-centric dataset collected via a field study. It
collected daily search logs as well as explicit feedback from 30 participants for one month.

• Testing set: not released yet.

Organizers
Yiqun Liu [yiqunliu@tsinghua.edu.cn] (Tsinghua University)
Jia Chen [chenjia0831@gmail.com] (Tsinghua University)
Fan Zhang [franky94@gmail.com] (Tsinghua University)
Jiaxin Mao [maojiaxin@gmail.com] (Renmin University of China)
Weizhi Ma [mawz@tsinghua.edu.cn] (Tsinghua University)Schedule⚐ July 1, 2021: Session Search registration due⚐ July 15, 2021: Dataset Release⚐ Sep 1, 2021 - Nov 30, 2021: Formal Run⚐ Dec, 2021 - Jan, 2022: Relevance Assessment⚐ Feb 1, 2022: Evaluation Result Release⚐ Feb 1, 2022: Draft Task Overview Paper Release⚐ Mar 1, 2022: Draft Participant Paper Submission Due⚐ May 1, 2022: All Camera-ready Paper Submission due⚐ Jun 2022: NTCIR-16 Conference & EVIA 2022 in NII, Tokyo, JapanReferences[1] Carterette, B., Kanoulas, E., Hall, M., & Clough, P. (2014). Overview of the TREC 2014 session track.
[2] Yang, G. H., & Soboroff, I. (2016). TREC 2016 Dynamic Domain Track Overview. In TREC.
[3] Zhang, F., Mao, J., Liu, Y., Ma, W., Zhang, M., & Ma, S. (2020, July). Cascade or Recency: Constructing Better Evaluation
Metrics for Session Search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development inInformation Retrieval (pp. 389-398).
[4] Chen, J., Mao, J., Liu, Y., Zhang, M., & Ma, S. (2019, November). TianGong-ST: A New Dataset with Large-scale Refined
Real-world Web Search Sessions. In Proceedings of the 28th ACM International Conference on Information and Knowledge
Management (pp. 2485-2488).
[5] Liu, M., Liu, Y., Mao, J., Luo, C., & Ma, S. (2018, June). Towards designing better session search evaluation metrics. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.

!ank you!

9