EVENT REPORT

Report on the 46th ACM SIGIR Conference
on Research and Development
in Information Retrieval (SIGIR 2023):
Reflections from the Program Co-Chairs
Makoto P. Kato

Josiane Mothe

University of Tsukuba
Japan

INSPE, Univ. Toulouse Jean-Jaurès, IRIT UMR5505
Francempkato@acm.org

Josiane.Mothe@irit.fr

Barbara Poblete
University of Chile, Chile
Amazon.com, USA
bpoblete@uchile.cl

Abstract
The ACM SIGIR Conference on Research and Development in Information Retrieval has
been experiencing significant growth over the past few years. In 2023, SIGIR received a
total of 822 full paper submissions and 1,787 papers/proposals. This has brought several
challenges to the conference program organization. This article reports some of the main
aspects of the work conducted by the program committee (PC) co-chairs for SIGIR 2023,
and discusses emergent issues of the review process and program design. We expect that
this document serve as a reference for future PC co-chars and as a starting point for further
discussions towards improving the review process and overall conference experience.
Date: 23–27 July 2023.
Website: https://sigir.org/sigir2023/.

1

Introduction

The international ACM SIGIR Conference on Research and Development in Information Retrieval
is a mature conference that in 2023 was on its 46th edition [Chen et al., 2023]. As PC co-chairs wewere honored to take on the challenge of this year’s organization. Now, having had a successful
run, we have put together this report with the purpose of documenting and reflecting on the workthat we did. On one hand, we expect this to be a first draft of a PC Chair’s handbook that willgrow in upcoming years. On the other hand, we wanted to take this opportunity to put out there
some of the discussions and thoughts that we had. Discussions that we believe should be opened
to the community, with the goal of improving in future conference editions.
ACM SIGIR Forum

1

Vol. 57 No. 2 – December 2023

The following report attempts to summarize some of this year’s features, such as schedule andbroadly characterize the received submissions. We also detail how we worked to expand the sizesof the senior program committee (SPC) and program committee (PC), in light of the growth insubmissions. We discuss the approach taken for desk rejection, which includes plagiarism. In
addition, we talk about the problems faced regarding conflicts of interest (COI) declaration anddetection, and how this influenced our approach to reviewer assignments.
In this year’s SIGIR conference, as well as in 2022, PC chairs oversaw the complete academic
conference program, which included all other tracks. This is, PC Chairs selected and invited track
chairs, as well as provided high level advice and supervision for each one of these tracks. In thisreport however, we focus only on the full paper track discussion.
Much of the work we did and the issues we experienced are common to the organization ofmost medium and large conferences. Nevertheless, we had to reach out to several past chairs
to obtain all the information we needed to keep the conference working consistently with prioreditions. Hence, our effort to put some of those things in this report. Furthermore, despite thatthe issues we described may be well known, many of these still require discussion and agreements.
Such as, for example, how to automatize better review assignment avoiding undeclared COIs that
occur when there are very close knit research communities working on highly similar topics. Also,
how to continue to grow systematically and reliably the size of the PC.
We also share in this report other conversations that we—the PC co-chairs—had and that are
more specific to the IR community. We feel that many of these are matters which should be openly
discussed. Most of these discussions can be found at the end of each section. In particular, we do
not expect to find the answers in this report, or to expose our personal points of view, but rather
to document our work and spark an initial, hopefully ongoing, conversation with our colleagues.

2

Organization Timeline

Table 1 shows the timeline of SIGIR 2023, particularly focusing on the PC co-chairs’ tasks. There
were three phases in this timeline: track design, PC member recruitment, and paper review. Thevery first task we did is to decide the schedule of paper submission and review. Since it is highly
dependent on the publisher, we first consulted with ACM and Sheridan about the paper submission timeline. We then decided track chairs of each track, namely, short paper, resource paper,
perspective paper, reproducibility paper, demo paper, tutorial, doctoral consortium, workshop,
SIRIP, proceedings, and best paper co-chairs. PC member recruitment is explained in detail in
Section 4. Desk rejection, paper assignment, and reviewing process are described in Sections 5, 6,
and 7, respectively. Section 3 provides some figures about submissions.

3

Submissions

Over the past ten years, the number of full-paper submissions received by SIGIR has more thandoubled. In the case of short papers it has grown by a factor 2.3. Figure 1 shows graphically thistrend.

ACM SIGIR Forum2

Vol. 57 No. 2 – December 2023

Table 1. Timeline of SIGIR 2023.

Date
March 11, 2022
May 4, 2022
May 31, 2022
May 31, 2022
June 15, 2022
June 15, 2022
July 15, 2022
July 25, 2022
July 31, 2022
October 20, 2022
October 31, 2022
January 24, 2023
January 31, 2023
February 1, 2023
February 6, 2023
February 10, 2023
February 11-16, 2023
March 2, 2023
March 7-16, 2023
March 20, 2023
March 23-29, 2023
April 4, 2023
April 26, 2023
July 23-27, 2023

3.1

Task
PC co-chair invitation from the general chairs.
Kick-off meeting with the general chairs.
Get the publication schedule from ACM and Sheridan.
Determine submission, review, and camera-ready deadlines for papers.
Discuss continuing/discontinuing new tracks.
Create a preliminary Call for Papers (CFP).
Invite chairs of the other tracks.
Inform the other track chairs and ask them to start to create a CFP.
Decide on the SPC and PC members.
Distribute CFP.
Invite SPC and PC members.
Full paper abstracts due.
Full papers due.
Desk-reject papers that violate the ACM policy or requirements.
Run automatic assignments & register reviewer requirements.
Paper assignments sent out; reviewing begins.
PC and SPC members perform sanity checks on papers assigned.
Reviews due.
PC and SPC discuss papers.
Final recommendation made by SPC (meta-reviews due).
SPC meetings with PC chairs.
Full paper notifications.
Camera-ready papers due.
SIGIR 2023 conference.

Authors and their countries

The number of unique authors has also drastically increased and their region/country affiliation
has evolved. In 2017 (Asia, Tokyo), there were 1,056 authors from 42 countries in the full paper
track: 299 from China, 232 from USA, and 525 from the rest of the world1 . In 2023 (Asia,
Taipei), also considering the papers submitted to the full paper track only, there is a total of 2,958
different authors (3 times more than in 2017) from which 1,743 from China (5.8 times more) and
354 from USA (See Figure 2.a). Most of the authors are affiliated to a Chinese university, lab ororganization (See Figure 2.b).
The number2 of full papers submitted (Figure 3.a) and accepted (Figure 3.b) shows China as
the leading source of contributions. The US follows in second place, but by a considerable margin.
1
2

https://dl.acm.org/doi/proceedings/10.1145/3077136
A paper that is co-published by authors in USA and in China appears twice.

ACM SIGIR Forum

3

Vol. 57 No. 2 – December 2023

Figure 1. The number of submissions over the last 10 years has more than doubled for the full-paper
track and has been multiplied by 2.3 for the short paper track. This information has been collected fromthe introductions to each year conference proceedings.

(a) per location(b) per continent

Figure 2. Number of distinct authors (a) per location – 10 most frequent locations (b) per continent(based on EasyChair self declaration).

3.2

Authors and submissions

There were 2,280 authors who submitted one paper and one author who submitted 19 papers in
the full paper track (See Figure 4).
Submissions topics were mainly centered on the topic of recommender systems (shown in
Figure 5.a). The conclusion is the same when considering the keywords authors associated totheir submissions (Figure 5.b).
The acceptance rate is slightly different according to the topic of the paper (See Figure 5.a).

ACM SIGIR Forum

4

Vol. 57 No. 2 – December 2023

(a) Submitted papers to the full track(b) Accepted papers to the full track

Figure 3. The number of submissions per location (based on the declarations in EasyChair paper
submission system. Scales on the Y-axes are not the same).

Figure 4. Number of submissions per author.

Topics have changed over years. In 2014 the most popular topics for accepted papers in the
full-paper track were : Document Representation and Content Analysis (13%), Queries and Query
Analysis (16%), Users and Interactive IR (17%) [Geva et al., 2014]3 . In 2017, the most popular
topics for accepted papers were ranking algorithms (19%), text representation (17%), behavioral
analysis (15%), recommender systems (14%), and user studies (14%) [Kando et al., 2017]4 . The
shift to recommender system being the most popular topic occurred in 2018. In 2023, the mostpopular topic was Search recommendation & content analysis for search and recommendation.
This topic was among the ones with the highest acceptance rate (See Figure 5.a).

3
4https://dl.acm.org/doi/proceedings/10.1145/2600428
https://dl.acm.org/doi/proceedings/10.1145/3077136

ACM SIGIR Forum

5

Vol. 57 No. 2 – December 2023

a) Topic of the paper as self-declared on easy chair topicsb) Topic of the paper based of the keywords the authors associated to their submission
Figure 5. Topics of the paper (a) as self-declared on EasyChair (b) most frequent keywords and theirfrequency.

4

PC Member Invitation

Next we discuss our approach for recruiting this year’s SPC and PC.

4.1

Enlarging PC

The increasing number of submissions (See Figure 1) as well as the growing contribution from Asia,
especially from China (See Figure 2), raise the question of geographical distribution of the PC
members. In parallel, in 2023 we wanted minorities in genre and location to be considered more.
We made a great effort to enlarge the program committee and to include more researchers from
Asia. We increased by 70% the number of PC members with an Asian affiliation (See Figure 6).
Our target was also to decrease the number of papers assigned to each PC member. It took theequivalent of about 3 weeks full time. The final PC was composed of 805 PC and 159 SPC5 .
5These numbers are based on the reviewer list in the proceedings. They include those who were invited after
the assignment and exclude those who gave no review.

ACM SIGIR Forum

6

Vol. 57 No. 2 – December 2023

Figure 6. PC member continent affiliation in 2022 and 2023. AM stands for Americas, AS for Asia andEU for Europe.

We put in place several means to enlarge the 2023 PC candidates:
1. We asked for previous years PC and SPC. We fuse the information from the 4 past years
and considered the status for each year (PC / SPC). The information on genre and location
was not uniformly provided; we completed that information.
2. We offered the possibility to self-nominate or to nominate colleagues or acquaintances. We
got 62 suggestions. We manually analyzed their Google Scholar and DBLP profiles.
3. We queried the Web of Science considering past authors from SIGIR; the authors and their
number of publications were collected. We kept the most frequent authors. We also manually
analyzed their Google Scholar and DBLP profiles as well as those of their co-authors.
4. Having in mind that the number of submissions on the topic of recommender systems isincreasing, we also considered adding more specialists of that domain.
Past PC members were included, but not necessarily in the same role. In (2) and (3), after
selecting the main authors (in terms of number of publications), we also focused on adding PC
who will raise to a more varied distribution in terms of locations (countries, continent, universities,
companies) and gender. For PhD students, we restricted ourselves to last-year PhD students, close
to defense, with good publication record in IR venues. For young researchers, we also restricted
to researchers who have published in SIGIR or related conferences. Regarding (4) we included
more PC specialists in this topic.
In addition, we kept a pool of emergency reviewers with the intention to call upon themwhen in need of missing reviews or additional reviews in case of low quality reviews. Emergency
reviewers were specifically invited for this role, some of them came from self-nominations, others
were experts on niche topics and we also invited reviewers who initially rejected our SPC/PC
invitation.
This year we assembled a single PC for both the full and short tracks.

ACM SIGIR Forum

7

Vol. 57 No. 2 – December 2023

Figure 7. Reviewing vs authoring effort. Ratio of the number of reviewers over the number of authors,ordered by the number of authors affiliated to the location (left), and by the ratio (right). The y-axes
have different scales

4.2

Senior and Program Committee Members

Our initial target was to balance the SPC in terms of location (1/3 Americas, 1/3 Europe andAfrica, 1/3 Asia and Oceania) and gender (get closer to 50/50). We also considered the affiliationof SPCs to avoid over-representing certain organizations. We did not systematically re-invited past
years’ SPCs into the same role. This is mainly because (1) we thought it is important to invite
new members into this role, and (2) we wanted it balanced in terms of organizations, locations
and gender.
We enlarged the number of SPCs. New SPCs were past PC (at least having been 3 or 4 times
PC, although some exceptions were made for less represented countries). Our target of genderbalance was not completely achieved, but we managed to reduce the gender gap further thanprevious years, and also include more researchers from different countries.

4.3

Authors vs Reviewers Ratio

Although we have tried to enlarge the PC and to modify the geographic distribution, we canobserve that the distributions of authors and reviewers are very different (See Figure 7). For
example, the ratio of the number of reviewers over the number of authors is 0.1 for China while
it is 5 for Greece. Europe puts the highest effort in reviewing as shown by these ratios. Whether
the ratio should be more distributed is an open question.

4.4

Open Challenges and Suggestions for Future Editions

 Distribution of PC members: should the distribution of reviewers follow the one of authors?
If so, should it be based on geographical regions, continents or countries? Should we consider
reviewer affiliation as a variable as well? What are the potential bias risks?
 Invitation of new PC members: should authors of accepted papers from previous years
be automatically invited to join the PC? After how many papers or how many years ofcontributing? What about authors who submitted papers?
 Underrepresented countries or geographical regions: what is the best way to incorporate
them without introducing significant disparities in criteria?

ACM SIGIR Forum

8

Vol. 57 No. 2 – December 2023

 Impact of adding more PC members on some topics: do we encourage more submissions on
a topic by enlarging the number of reviewers on that topic?
 The PC was the same for long and short papers. It is crucial to make that very clear to
PC members, because bids and assignment to papers run separately. This may cause sometrouble due to being apparently asked for the same thing twice.
 Tools: for handling PC and SPC, we mainly used spreadsheets and EasyChair, which is not
fully appropriate for tracking and sharing the information. For example, it is not easy toshare invitation, reasons for which the invitees declined, etc. It could be interesting also tokeep track of more information.

SIGIR’s DEI committee could provide input and guidance to establish consistency in the diversity
and inclusion criteria for the PC and SPC. Currently, most of the aspects in the choice of PCs
and SPCs differ from one year to the next depending on the PC chairs serving at the time. Thiscan be an interesting way of integrating different perspectives, but some decisions may require alonger-term policy.

5

Desk Rejection

This year’s desk-rejection criteria followed the same policy as previous years, without changes. The
complete guidelines were included in the call for full papers and posted in the conference website6 .
The two main aspects considered in this policy were 1) failure to meet formatting guidelines and2) behavior that generated suspicions of academic dishonesty. Authors of desk-rejected papers
were notified and their papers removed from the reviewer pool before sending out PC and SPC
assignments.

5.1

Format Related Rejections

We reviewed all of the submitted papers to ensure they met formatting guidelines such as page
limit and anonymity. To help in our reviewing of formatting requirements, we automatically
pre-processed submissions to extract relevant features. From this we generated a report that
included information about each paper, and whether they may not meet some of the formatting
requirements. Any paper that appeared to be in violation of formatting restrictions was manually
verified by PC Chairs for deciding desk-rejection.
However, in the case of very minimal formatting mistakes some flexibility was allowed inagreement with all PC co-chairs. This was the case for papers in which the content exceeded thepage limit by a very small amount and that could have easily been fit in the current page limit,
without need of removing content. Three papers with minimal formatting mistakes were required
to be updated and not rejected due to the formatting problems.
Overall, at the end of this process, ten papers were desk-rejected due to important format
related problems and authors were informed before starting the review process.
6

https://sigir.org/sigir2023/submit/call-for-full-papers/

ACM SIGIR Forum

9

Vol. 57 No. 2 – December 2023

5.2

Lack of Topical Fit

As part of the desk rejection policy submission were also reviewed semi-automatically for candidates for lack of topical fit. This meant flagging papers that did not appear to be in the scope of
SIGIR to then have all PC chairs review them manually. Only 2 papers were rejected using thiscriterion.

5.3

Academic Dishonesty

Any form of academic fraud or dishonesty was a cause of desk rejection. However, as PC chairs
we only actively investigated content plagiarism and simultaneous submissions.
5.3.1

Content Plagiarism

Plagiarism in submitted papers was initially detected using iThenticate7 . Following this, only thosemanuscripts that met specific criteria were further scrutinized manually. As of 2023, iThenticate
was capable of identifying duplicates with various documents published online. However, a highduplication rate was not solely indicative of plagiarism; it could also include overlaps with papers
submitted by the authors to arXiv. Thus, plagiarism could not be determined based solely on
duplication percentage. Consequently, when a paper exceeded the duplication threshold, a PC
chair reviewed each case, verifying if the duplicated content was previously presented at otherconferences or journals (self-plagiarism) or contained multiple sentences from other papers.
Out of 228 papers reviewed manually, 11 were suspected of plagiarism. These papers underwent a subsequent review by all the PC chairs. Ultimately, nine papers were rejected for thefollowing reasons: seven papers due to plagiarism and two papers due to self-plagiarism. The
plagiarized content included material already presented at international conferences or journals,
as well as content from other authors’ papers submitted to arXiv. The sources of self-plagiarism
were primarily papers previously published in international journals.
A reevaluation for plagiarism was conducted on all accepted papers, utilizing iThenticate forindividual examination. There was no plagiarism found in the accepted papers.
5.3.2

Simultaneous Submission

In ACM’s policy, simultaneous submission is explained as follows:
The ACM does not normally permit manuscripts under review in its journals or conference proceedings to be simultaneously under review for another publication. (By”under review” we mean a manuscript that has been submitted, and has not been either withdrawn or rejected.) ... Under no circumstances shall a paper (or substantiallythe same paper) be simultaneously submitted to two or more publications, or to a second publication while still under review elsewhere, without a letter of notification to
the Editor-in-Chief (EiC) or Program Chair (PC) of each affected publication. Failure
to adhere to this policy is cause for rejection of the manuscript. Repeated violations
may lead to a ban on future submissions at the discretion of the EiC or PC.
7

https://www.ithenticate.com/

ACM SIGIR Forum

10

Vol. 57 No. 2 – December 2023

Guided by advice from past PC chairs, we contacted PC chairs of other international conferences with submission deadlines close to SIGIR 2023 and with similar topical focus, requesting
the sharing of submitted paper titles, authors, and abstracts. An automated calculation of abstract duplication rates was conducted, and papers with particularly high duplication rates weremanually scrutinized. In total, we identified four SIGIR 2023 papers with significant similarities
to submissions at other conferences. Based on mutual agreement with the other conferences’ PC
chairs, papers submitted to both venues were rejected. Note that papers withdrawn from theother conference were not considered in this process.

5.4

Open Challenges and Suggestions for Future Editions

Following editions of SIGIR could require authors to go through a formatting checklist at submission time. For example, by adding a step in the submission process with a form that includes one
checkbox for each formatting requirement.
Another suggestion is that it may be useful to have an open discussion on the desired flexibility
for paper topical fit. In our experience, there was not always agreement among PC co-chairs ofwhether a paper was within SIGIR scope or not.

6

Paper Assignment

6.1

Conflict of Interest

Before the bidding process, authors, PC and SPC members were asked to indicate their conflictsof interest (COIs) with authors. COIs were indicated to be those defined by ACM and detailed
in their policy8 .
To help COI declaration, we extracted the list of the co-authors from major conferences andjournals in information retrieval and provided this list as a spreadsheet file were each author
and PC member could search their names and view their co-authors. This approach allowed us toexpand COI detection, but was far from perfect as it did not allow us to detect people who work inthe same research groups as we discuss in Section 7. EasyChair did not allow us to automaticallyintegrate these COIs in the system. We also used the EasyChair automatic detection of COIs.
Although many reviewers declared their COIs, an important percentage did not.

6.2

Bidding

The bidding process for paper reviews was conducted from January 26 to February 2. The data
revealed that, on average, PC members bid (either “yes” or “maybe”) on 19.4 papers, with a
median of 13 papers per member. In contrast, Senior PC members showed a slightly higher
engagement, bidding on an average of 21.6 papers and a median of 17 papers. Notably, there wasa subset of members who did not participate in the bidding process. Specifically, 32 PC members
and 3 senior PC members abstained from bidding.
8

https://www.acm.org/publications/policies/conflict-of-interest

ACM SIGIR Forum

11

Vol. 57 No. 2 – December 2023

Furthermore, the bidding activity varied significantly among the members. Although there
was a request for members to bid on more than 20 papers, only 28.2% of PC members and 44.1%
of senior PC members met this threshold.

6.3

Assignments

We addressed the issue of friendly reviewers observed in other academic conferences. This issue
arises when authors solicit favorable reviews from acquaintances by bidding for them to review theirpapers. To mitigate this, we implemented an assignment restriction for the number of reviewers
from the same country/region as declared by the authors to one or fewer. This approach isbased on the assumption that friendly reviewers are likely to be from the same country/region
as the authors, though it does not completely resolve the issue. By geographically diversifying
the reviewers, we aimed to broaden the range of perspectives on each paper, thereby diluting thepotential influence of friendly reviewers.
Our submission and review processes were handled by EasyChair, and while its automatic
paper assignment feature was available, incorporating the aforementioned constraint was notstraightforward. Consequently, we developed a custom script to handle paper assignments outsideof EasyChair, later importing the results into the system9 .
The problem of paper assignment, exemplified by full papers, can be formulated as an integer
programming problem:

maximizen X
mXwij xij

i=1 j=1

subject toxij ∈ {0, 1} (i = 1, . . . , n; j = 1, . . . , m)nXxij = 4 (j = 1, . . . , m)
i=1

4≤

m
X

xij ≤ 5 (i = 1, . . . , n)jnXcij xij ≤ 1 (j = 1, . . . , m)

(1)

i=1

where xij = 1 indicates the j-th paper is assigned to the i-th reviewer, wij is a variable indicating
the bid to the j-th paper given by the i-th reviewer (with values: 1,005 for “yes”, 1,000 for“maybe”, 0 for “no”, and −100, 000 for COI), and cij is a binary variable indicating whether
the i-th reviewer is from the same country/region as any author of the j-th paper. The second
constraint ensures that each paper is reviewed by exactly four reviewers, while the third constraint
ensures that each reviewer is assigned between 4 and 5 papers. The final constraint limits thenumber of reviewers from the same country/region as the authors of each paper to at most one.
9

https://github.com/mpkato/SIGIR2023_paper_assignment

ACM SIGIR Forum

12

Vol. 57 No. 2 – December 2023

Table 2. Paper assignment statistics (PC).

Table 3. Paper assignment statistics (SPC).

MetricMetric# of reviewers# of reviewers with 1 paper# of reviewers with 2 papers# of reviewers with 3 papers# of reviewers with 4 papers# of reviewers with 5 papers# of papers with 0 willing reviewer# of papers with 1 willing reviewer# of papers with 2 willing reviewers# of papers with 3 willing reviewers# of papers with 4 willing reviewers# of reviewers with 0 bid paper# of reviewers with 1 bid paper# of reviewers with 2 bid papers# of reviewers with 3 bid papers# of reviewers with 4 bid papers# of reviewers with 5 bid papers

Count

Count

# of reviewers# of reviewers with 4 papers# of reviewers with 5 papers# of reviewers with 6 papers# of papers with 0 willing reviewer# of papers with 1 willing reviewer# of reviewers with 0 bid paper# of reviewers with 1 bid paper# of reviewers with 2 bid papers# of reviewers with 3 bid papers# of reviewers with 4 bid papers# of reviewers with 5 bid papers# of reviewers with 6 bid papers

797
1
16
23
593
164
6
17
36
83
682
38
11
23
47
522
156

160
4
128
28
155
669
4
1
3
10
94
32
16

Tables 2 and 3 show the statistics of the PC and SPC paper assignment in SIGIR 2023. While
92.8% of papers (765 out of 824) were assigned to three or four willing PC members, definedas those who bid for the paper, 81.1% of papers (669 out of 824) were assigned to willing SPC
members. Most of the SPC members were required to review one or two papers that they did not
bid for. This may suggest that the expertise diversity in SPC should be improved to cover more
submissions to the SIGIR conference.

6.4

Open Challenges and Suggestions for Future Editions

 With regard to COIs: should we explore the possibility of removing from the PC and SPC
anyone who fails to declare their COIs? This could improve the low compliance, however it
might also reduce considerably the size of the PC which always needs to grow.
 Updating authors and PC members on EasyChair: several authors and PC members wereregistered using outdated affiliations and emails in EasyChair, which did not allow to identify
correctly COIs with people who now work in their current organizations. Hence, authors andPC members could be asked to disclose their past and present work emails and affiliations.
 Bidding: should it be reasonable to remove the PC and SPC anyone who fails to participate
in the paper bidding? No bidding might be a sign of low engagement in the review process,and resulted in random paper assignment, potentially increasing the review load. WSDM
implemented this.

ACM SIGIR Forum

13

Vol. 57 No. 2 – December 2023

Figure 8. Number of reviews in, sorted by dates. Deadline was 03 March 23; bars in blue are before
that deadline, red after.

7

Reviewing Process

We describe different aspects of the reviewing process, including the guidelines that were provided
to PC and SPC members, review submission, and decision agreement.
Every submission was assigned a senior program committee member (SPC) and at least threeprogram committee members (PCs). Since our program committee had grown considerably, we
attempted to initially assign 4 PCs to each submission. This, based on our past experience inother conferences where many reviews were not sent in reasonable time of review deadline. On
average each PC member was assigned 4 papers and each SPC 5 papers to review.

7.1

Reviewing Guidelines

PCs and SPCs were informed of their roles and reviewing guidelines by email and through a“reviewer instructions” document. This document was based on “Reviewing Guidelines for SPC
and PC Members” in SIGIR’2110 with some minor revisions that we added. Reviewer guidelines
elaborate on each of the different criteria for which submissions needed to be evaluated, including
relevance, originality, technical soundness, presentation, reproducibility, strengths, weaknesses,
overall recommendation, and detailed comments, among others. The complete reviewer guidelines
can be found in Appendix A.
An important aspect of the reviewing process is that we explicitly indicated that both PC and
SPCs are expected to read all of their assigned papers. In the case of SPC members, these are
required to form their own opinion of the work. In the case of PC members, even in the case ofusing sub reviewers the original PC member is ultimately responsible for the review and should
form their own opinion of the work and include that in their review. The goal of this is to add
quality checks for reviews and for submissions at different stages of the reviewing process.

ACM SIGIR Forum

14

Vol. 57 No. 2 – December 2023

7.2

Collecting Reviews and Recommendations

For the full paper track, we received 3,100 reviews. By the time of the reviewing deadline, only
half of these had been submitted (See Figure 8). To obtain missing reviews our SPCs reached
out personally to PC members with outstanding reviews and attempted to reach an agreement ofwhen the review would be submitted. In many cases we reached out personally as PC chairs after
several unsuccessful attempts had been made to certain reviewers by the SPCs. We iterated thisprocess until papers had at least 3 reviews.
During this period SPC members also attempted to foster discussions with the PC membersof each paper. The goal of this discussion was to see if it was possible to reach an agreement onwhether the paper met the quality bar for SIGIR. If an agreement was not possible, the discussion
was expected to provide sufficient context to support the SPCs recommendation.
In several cases, when there was a large disagreement between PC members, a recommendation
was made by the SPC using their expert judgment, if they felt confident in the specific area ofthe paper. Otherwise, an additional emergency reviewer with expertise on the topic was invited.

7.3

PC Chairs & SPC Meeting

Once all reviews and recommendations were collected, we examined each of them and flagged
papers for which additional discussion was required. We then held meetings to review recommendations and finalize decisions, with each one of the SPC members who had any papers thatneeded additional discussion (71 out of 159). If relevant to the final decision, the SPC was asked
to add the relevant points of their meeting with the PC chairs. This was done with the purposeof conveying all relevant information to authors on why decisions were made on their papers.
In very few cases, papers with relatively good scores were not accepted. The main reason waswhen the SPC being an expert in the topic brought up valid concerns to the PC chairs about thequality of the work. In some of these cases, we found overly positive reviews which were calibrated
during the SPC meeting. However, in a couple of cases we found that the highly positive review
came from a reviewer that had an undeclared COI with the authors. In particular this occurred
when authors or reviewers who were coauthors, or who worked in the same research group, but
used an outdated email address/affiliation in EasyChair.
We note that this year there was enough room to accept all of the papers that met the SIGIR
acceptance bar. In that regard there were no papers that were left out due to a cutoff percentage
or score.

7.4

Conference Program

Before working on the program, we asked the authors whether they planed to present their paper
remotely or in person.
Regarding the program itself, initially, we thought that automatic classification algorithms
could help us produce a comprehensive program; grouping papers into sessions. After a few trials,
we decided to apply a more manual approach, grouping the papers according to their topics into
groups of 5 to 7. For topics with a large number of papers, we created coherent sub-groups and10

https://sigir.org/sigir2021/calls-long-guidelines/

ACM SIGIR Forum

15

Vol. 57 No. 2 – December 2023

scheduled them in sequential (but not parallel) [Chen et al., 2023]. Moreover, we arranged sessions
so that when parallel, some would start with in-person presentations and finish with remote whileothers would be the other way around. At the end, many authors could not attend the conferencein person but that information was given after the final program was set.

7.5

Open Challenges and Suggestions for Future Editions

 Transparency on reviews: this year, we asked PC members whether they agreed in sharing
their reviews with the SIGIR Asian chapter. This is clearly a first step in more transparency
in reviews. We may lack of an agreement in sharing more data on reviews such as lateand missing reviews. This type of information could be useful for next years chairs, but
information cannot be shared without a prior agreement. This should be anticipated so that
reviewers clearly acknowledge on chairs sharing this kind of information.
 In the case of undeclared COIs, it is hard to say that there was bad intent, in particularwhen using outdated affiliations. However, we might want to have some incentive or penalty
for failing to disclose all of the necessary information to detect COIs. For example, failing to
declare COIs with someone in your same research group or who you have published actively
with.
 Regarding unresponsive reviewers who failed to submit any reviews and to communicate
their reasons with the PC chairs, we wonder if this information should be registered for
future editions of the conference. We are not sure what is the best practice, but if that werethe case then it should be informed and consented upon, as part of the agreement when
accepting to be a PC member.
 Also, consent from the PC and SPC might also be wanted to potentially analyze anonymously
reviews and decisions for each of the conference editions.

8

Conclusion

This report includes a series of open questions and discussions. For many of them, there arequestions PC chairs have to answer yearly and with local answers. Some of these questions may
deserve a full attention of the IR community and possible long term answers should be given.

Acknowledgments
We extend our deepest gratitude to all the SIGIR 2023 organizers for their tremendous efforts
in the conference organization. We also would like to thank all the track chairs. We would liketo express our sincere appreciation to the reviewers for their expert insights and constructive
feedback, and the authors for their contributions and commitment to the conference program.

ACM SIGIR Forum

16

Vol. 57 No. 2 – December 2023

References
Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, andBarbara Poblete, editors. SIGIR ’23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, 2023.
Association for Computing Machinery. ISBN 9781450394086.
Shlomo Geva, Andrew Trotman, Peter Bruza, Charles L. A. Clarke, and Kalervo Järvelin, editors. SIGIR ’14: Proceedings of the 37th International ACM SIGIR Conference on Research &Development in Information Retrieval, New York, NY, USA, 2014. Association for Computing
Machinery. ISBN 9781450322577.
Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors.
SIGIR ’17: Proceedings of the 40th International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, New York, NY, USA, 2017. Association for Computing
Machinery. ISBN 9781450350228.

A

Reviewing Guidelines for SPC and PC Members

Every submission is assigned a Senior Program Committee member (SPC) and at least three
Program Committee members (PCs).
 As an SPC member, you are responsible for ensuring the overall quality of the review
process. As reviews come in, you should encourage reviewers to provide constructive andinformative reviews, flag any problems, and let the PC Chairs know if an extra emergency
review is needed. After the review deadline, you must lead discussions for your assigned
papers. Based on the discussion, you will write a meta-review, and make a final recommendation for acceptance/rejection. This recommended decision will be passed on to the PC
Chairs and used as input to the Program Committee Meeting. Note that you are expected
to personally read and build your own opinion of every paper that is not a clear reject. Moredetails on your tasks are included below.
 As a PC member, your job is to provide critical assessments of the papers assigned to you,based on the review form that we have prepared (see below). For each paper assigned toyou, your and your fellow reviewers’ feedback will form the basis for a discussion. Note thatyou must conduct each review personally. In other words, you should not delegate any ofyour reviews. If you want to request help from a colleague or student, this is allowed only
in well-justified cases, but note that you still must read the paper, provide a review andparticipate in the discussion. More details on your tasks are included below.

A.1

SPC Member Tasks

As an SPC member in the primary SPC role, you are responsible monitoring the review process of
PC members, conducting quality checks of the reviews, sending reminders to PC members whennecessary, leading discussions in the discussion phase trying to reach consensus, writing a meta
ACM SIGIR Forum

17

Vol. 57 No. 2 – December 2023

review, and making the final recommendation of accept/reject. Please inform us of any difficulties
as soon as possible at any stage.
Paper Review Management: You will be the main person to manage the papers assigned toyou. Flag missing reviews close to review deadline and inform PC chairs.
Read Each Paper: Note that you are expected to personally read and build your own opinionof every paper that is not a clear reject
Review Quality Checking: You are in charge of ensuring that the review reports reach the
expected quality. You should ask the PC members to rewrite or modify a review if its quality is
insufficient.
Discussions: You are responsible for initiating and leading the discussions as needed.
Recommendation: You will write a meta review, and make an accept/reject recommendation to
the PC chairs. This also includes participating in the PC meeting to finalize decisions if required
to do so.

A.2

PC Member Tasks

As a PC member, your first task is to conduct objective and thorough reviews in a timely manner,answering all the questions included in the review form:
 Relevance to SIGIR: the relevance of this paper to SIGIR, noting that SIGIR’s scope isbroad.
 Originality of the work: how original the work described in this paper is.
 Technical soundness: quality of the technical content of this paper.
 Quality of presentation: the clarity of the presentation of this paper.
 Adequacy of citations: how the paper discusses and compares with related work.
 Reproducibility of methods: whether researchers can reproduce the methods and results
described in the paper with reasonable efforts.
 Strengths: describe 3 strengths of the paper.
 Weaknesses: describe 3 weaknesses of the paper.
 Overall recommendation
 Detailed comments to authors: provide a detailed review for the authors, including
justifications for your scores and overall recommendation.
 Nominate for Best Paper: whether this paper might be considered as a best paper
candidate.
 Nominate for Best Student Paper: only applies to student papers Reviewer’s confidence.
 [NEW] Review transfer to SIGIR-AP 2023. Let us know if you authorize us to transfer
your review to SIGIR-AP 2023 PC chairs (http://www.sigir-ap.org/). Authorized reviews
will be used if the paper is rejected at SIGIR 2023 and the authors choose to resubmit it to
SIGIR-AP 2023. Reviewer identity will be strictly protected in this process.

You should assign a score for each of these questions. In the text area of “Detailed comments
to the authors”, and supply a detailed rationale for these scores. Allow us to remind you of thebasic guiding principles of good reviewing.
You are not allowed to delegate the review task to another person. You are responsible toprovide the review. It is however possible to involve a sub-reviewer (a senior student or a colleague)
ACM SIGIR Forum

18

Vol. 57 No. 2 – December 2023

as part of the training, with the agreement of the SPC. Even in this case, the PC should write
his/her own review that represents his/her judgments.
Your review is not just a vote for acceptance/rejection; it is an essential input to a discussion
among the reviewers and to the Program Chairs. You are assisting your fellow PC members, theSPCs, and the Program Chairs by providing arguments for or against acceptance. In some cases,
there will be divergence among reviewers’ ratings of the paper; if you provide only a rating andterse summary, without an adequate rationale, it will not be helpful.
Your review also serves as constructive input to the authors to help them understand theoutcome of the review process and improve their work for future venues. Please be helpful andpolite in your comments. The first author may be a student – think about what kind of feedback
would have been helpful when you were starting in the field.
Start your review with an assessment of what you consider to be the main contribution of thepaper. Please do not just repeat what the authors say they did. You should provide your own
summary of what you gained by reading the paper.
Whether you like or dislike a paper, please say so in a manner that is courteous and helpful
to the authors and informative to your SPC and Program Chairs. You will be asked to rewrite
reviews that do not meet this expectation.
In the text box labeled “Overall recommendation”, please summarize your main points. It
is important to point out weaknesses and validity issues, but it is equally important to identify
the contribution of a submission. Ultimately, a submission’s acceptance depends on its novel
contribution, not perfection. Note again that we are looking for an evaluation of the paper, notjust a recommendation.
We would like to draw your attention to some of the reviewing criteria.
 Relevance: We ask the authors to identify one or more topics when submitting a paper.
The Relevance criterion should not be evaluated solely with respect to the topics identified,
but with respect to the conference. We encourage the PC members to be open on thisquestion: if the work is related broadly to IR, or it is potentially useful to IR (assuming that
this is described in the paper), then the paper should be considered to be within the scopeof SIGIR. If you find that a paper is fully outside of the scope of SIGIR you can notify the
SPC Chairs as we might consider a desk rejection.
 Originality: We encourage authors to take risks and tackle fresh and diverse researchchallenges that will interest our research community. Reviewers should give preference toan intriguing paper whose research might not be fully completed (as long as the style isgood, the technical approach is sound and main claims are supported), over a paper whose
research is perfectly addressed but is only incremental.
 Adequacy of citations: One issue that may arise is that authors miss some of the priorresearch that has been published in the area. This should be regarded as being a fatal flaw
only if the missing work critically affects their conclusions; remember, authors will havethe opportunity to make small editorial changes to their papers (including adding missing
references) if they are accepted. If you do regard a paper as being unacceptable because oflack of reference to prior work, you should supply sufficient detail about this prior researchin the form of a complete reference, ideally including its DOI, so that the authors canunderstand why you believe their paper should not be accepted. We encourage you toprovide this level of detail for all references that you consider missing in the submission.

ACM SIGIR Forum

19

Vol. 57 No. 2 – December 2023

 Reproducibility of methods: This question is not about the use of proprietary data. It is
about whether you think the authors provide sufficient details to reproduce the work. If theauthors make their dataset and code available, this certainly helps to reproduce the results;
but the reproducibility criterion is not limited to the sharing of data and code. Do you thinkthat other researchers would be able to reproduce the method and/or results presented in
the paper if they had access to the same or similar resources? Are the descriptions of
the methods used detailed and accurate? Are all important technical details described?
Given the resources used in the paper, or (if they are unavailable) similar resources, could
researchers carry out similar experiments to verify the results? What further description
could the authors provide?
 Double submission or plagiarism: Pay special attention to this! If you are aware thatthe submitted paper is under review for another conference or journal, or is very similar tosuch a submission or published paper, please clearly put this in your comments. Program
Chairs will double check the case and sanctions will be imposed on any verified offending
authors. Note that workshop papers and other non-archival publications such as arXiv are
not considered double submission or self-plagiarism.
 Anonymity: The submissions should be anonymous. If the identity of the authors are
revealed in the paper, please state it clearly in your comment. However, do not try to guess
who the authors could be from the dataset used in the experiments – using a dataset froma company does not necessarily mean that the authors come from the same company. So,
mentioning a dataset or experimental environment from a company or academic institution
is not considered as a violation to the anonymity of submission.

ACM SIGIR Forum

20

Vol. 57 No. 2 – December 2023