EVENT REPORT

Report on the 9th ACM SIGIR / the 13thInternational Conference on the Theory ofInformation Retrieval (ICTIR 2023)
Masaharu Yoshioka

Mohammad Aliannejadi

Julia Kiseleva

Hokkaido UniversityJapan

University of Amsterdam
The Netherlands

MicrosoftUSA

yoshioka@ist.hokudai.ac.jp

m.aliannejadi@uva.nl

julia.kiseleva@microsoft.com

Abstract
The 9th ACM SIGIR / The 13th International Conference on the Theory of Information Retrieval (ICTIR 2023) held in Taiwan co-located with SIGIR 2023. This brief report provides
an overview of ICTIR 2023 and introduces the student reviewer program, which provides anopportunity for the students to understand and contribute to the conference review process.
Date: 23 July 2023.
Website: https://sigir.org/ictir2023/.

1Introduction

The International Conference on the Theory of Information Retrieval (ICTIR) is the premier conference on theoretical information retrieval. ICTIR focuses primarily on the theoretical aspects ofIR and interdisciplinary research that links IR to adjacent fields like artificial intelligence, cognitive science, digital humanities, machine learning, natural language processing, recommendationsystems, and social computing. ICTIR also welcomes papers that aim to define novel tasks orapply fundamentally different ideas to information retrieval tasks.
The 9th ACM SIGIR / The 13th International Conference on the Theory of Information
Retrieval (ICTIR 2023) held in Taiwan co-located with SIGIR 2023. ICTIR was originally held
as an independent conference and changed its style as a co-located event of SIGIR since 2021. We
would like to express our sincere gratitude to SIGIR 2023 local organizers.
ICTIR 2023 was a one-day conference held in a hybrid format. To avoid any schedulingconflicts with SIGIR’s main conference sessions, we decided to use two parallel tracks this year.
We encourage in-person participation but are accepting hybrid presentations due to visa issues.

2Program

For the one-day program, we had a keynote speech entitled ”Evaluating Parrots and SociopathicLiars” [Sakai, 2023] by Professor Tetsuya Sakai and 30 oral paper presentations selected fromACM SIGIR Forum

1Vol. 57 No. 2 – December 2023

(a) Submitted(b) AcceptedFigure 1. Word cloud of the titles of the (a) submitted and (b) accepted papers.

73 high-quality submissions (acceptance rate of 41%). Figure 1 shows word clouds of the titlesof the submitted and accepted papers. We have chosen “Perspectives on Large Language Models for Relevance Judgment” [Faggioli et al., 2023] as the best paper and granted “HierarchicalTransformer-based Query by Multiple Documents” [Huang et al., 2023] an honorable mention.

3

Student Reviewer ProgramThis year, we conducted an experiment inviting students to volunteer and help in the review
process. To do so, we invited the students through an open call from all over the world and asked
them to volunteer to review papers. They had to fill out a form with some basic information
about their experience in publishing papers in peer-reviewed conferences and/or journals, as wellas their experience in reviewing papers. We received a variety of applications to join the PC, andadopted an inclusive approach, admitting almost all the applicants. We received 86 applications,from which we invited 80 students. This led to an addition of 68 student reviewers. Figure 2
shows the distribution of the applications in terms of their education, where we see that most of
the invited volunteers are PhD students.

3.1

Paper Assignment

Our goal was to keep the review load of the students as low as possible (1–2 assigned papers perstudent). To avoid an unwanted negative impact on the review quality, we considered the student
reviewers as additional reviewers for each paper. Therefore, we assigned three main reviewers toeach paper with an addition of 1–2 student reviewers.

ACM SIGIR Forum

2Vol. 57 No. 2 – December 2023

Table 1. Reviews and PC statistics.

Non-student

Student# reviewers# reviews# comments48195
320

6877
41

Avg. review length
Avg. review number
Avg. review score

425
2.64
0.07

533
2.23
0.31

StatusRecent PhD graduate
MSc student
Senior PhD student (3+ years)
Junior PhD student (1-2 years)
05

10

1520
Invited

2530

35

1750
1500
1250
1000
750
500
250
05Review number# review tokens

Figure 2. Distribution of invited student reviewers’ status.

Non-student

43
21Student(a) Review LengthNon-student

Student(b) Review Number

Figure 3. Comparison between non-student and student reviewers in terms of (a) review length and (b)review number (as indicated by EasyChair).
3Review score

2101
Decision
ACCEPTREJECT

2
3
Non-student

Student

Figure 4. Reviewer score distribution for accepted and rejected papers, per student and non-student
reviewers.

ACM SIGIR Forum

3

Vol. 57 No. 2 – December 2023

Table 2. Review score confusion matrix of non-student and student reviewers.

AcceptRejectNon-student

Student

Non-student

Student65
2125
15

1795

731

AcceptReject

0.4

0.4

0.3

0.3

0.20.20.1

0.1

0.0

42 02
Review score

0.04(a) Non-student

4202
Review score

4(b) StudentFigure 5. Review score distribution of (a) non-student and (b) student reviews.

3.2Analysis

As part of this program, we aimed to understand if students exhibit a different behavior whenreviewing papers, and whether a more inclusive approach of inviting junior students as PC can bebeneficial. First, in Table 1 we report some general statistics of the reviewers and their reviews.
As we see, the students wrote significantly fewer reviews in total, as we aimed to limit the student
reviews to 1–2 reviews per paper. However, we see that they tended to write longer reviews. Thisis more evident in Figure 3a where we plot the distribution of review tokens for non-student andstudent reviewers, suggesting that the students can dive deeper into the technical details of thepaper and bring new perspectives that more senior reviewers do not necessarily focus on. We
also look at the average review number as an indicator of punctuality, where we see in Figure 3bthat the student reviewers did indeed submit their reviews faster than non-student reviewers. Interms of discussion, however, we did not observe a great contribution. Next, we focus on the score
distribution and their agreement with the final decisions. We see in Figure 4 for accepted papers,that the score distribution of non-student and student reviewers is very similar. For rejected
papers, on the other hand, we observe that the students tend to give higher scores. Looking closer
at the confusion matrix of the scores in Table 2 we see higher false negatives for student review
scores, especially on the rejected papers. This is also corroborated in Figure 5 where we plot thescore distributions and see that there is a clear positivity bias in the student scores, comparedto the non-student reviewers. We think that a potential reason for this bias is the fact that thestudents were mostly assigned one paper to review, which leaves out the possibility of comparing
the quality of different papers in the review pool.

ACM SIGIR Forum4Vol. 57 No. 2 – December 2023

4Summary

We briefly introduce ICTIR 2023 and our new experimental trial of the student review program.
We think this program can provide a good opportunity for the students who did not have any
chance to understand the review process from the reviewer side and shed light on the potential
influence they can have in the program of future conferences. We conclude that even though thestudent reviews can be less critical, they will provide an additional perspective to the reviews andcan potentially spark interesting discussions about the papers. ICTIR 2024 will be held co-locatedwith SIGIR 2024. We welcome submissions and participation in the conference.

Acknowledgement
We would like to thank Prof. Atsushi Keyaki for his support as publication and publicity chair.
We also would like to SIGIR 2023 local organizers for their support to run the conference.

References
Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen,
Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning
Wachsmuth. Perspectives on large language models for relevance judgment. In Proceedings ofthe 2023 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR’23, page 39–50, New York, NY, USA, 2023. Association for Computing Machinery. ISBN9798400700736. doi: 10.1145/3578337.3605136. URL https://doi.org/10.1145/3578337.36
05136.
Zhiqi Huang, Shahrzad Naseri, Hamed Bonab, Sheikh Muhammad Sarwar, and James Allan.
Hierarchical transformer-based query by multiple documents. In Proceedings of the 2023 ACM
SIGIR International Conference on Theory of Information Retrieval, ICTIR ’23, page 105–115,New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700736. doi:
10.1145/3578337.3605130. URL https://doi.org/10.1145/3578337.3605130.
Tetsuya Sakai. Evaluating parrots and sociopathic liars (keynote). In Proceedings of the 2023
ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’23, page 1,New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700736. doi:
10.1145/3578337.3605144. URL https://doi.org/10.1145/3578337.3605144.

ACM SIGIR Forum5Vol. 57 No. 2 – December 2023