SlowGuess/NAACL_2022_OCR huggingface dataset:
We conclude that our sample does provide evidence (the first, to our knowledge) that researchers in interdisciplinary fields who perform different kinds of research may have differing preferences for what information should be taken into account for paper-reviewer assignments. If that effect is robust, it should be considered in assignment systems for interdisciplinary fields. We hope that this finding would be explored in a larger study, taking into account both the experience of authoring a given type of paper and how central that type of research is for a given researcher (a factor that we did
not consider). Another direction for future work is exploring this question from the perspective of demographic characteristics and the type of institution the respondents work in. Should there be significant differences, more targeted assignments could be a powerful tool for diversifying the field.
5.3 Ideal workload
We asked our reviewer and AC respondents how many assignments they received at their most recent NLP venue, and what would be the optimal number (given a month to review, and a week for AC assignments). For ACs, the mean optimal number of assignments is  $8.5 \pm 4.2$  vs.  $9.1 \pm 5.1$  they received at the most recent venue, and for reviewers it is  $2.8 \pm 1.0$  vs.  $3.3 \pm 1.8$ . Whether this is an
issue depends on how much time a given venue allows. The ARR reviewers have even less than a month, and they indicated preference for fewer assignments than they received  $(2.4 \pm 1.0$  vs  $3.3 \pm 1.9)$ . See App. fig. 10 for data on other venues.
The lack of reviewers is a well-known problem. One of the possible causes is that many authors are students not yet ready to be reviewers. To investigate that, we asked the authors if they also reviewed for the venues where they last submitted a paper, and the reviewers and ACs - if they also submitted. If the core problem is that many authors are not qualified, we would expect more non-student authors to also be reviewers. Among all respondents there are  $24\%$  authors who submit to a venue but do not review there or help in some other role (fig. 1), but if we consider only non-student respondents that ratio is still  $18\%$  (see non-student role distribution in App. fig. 9). This suggests that many qualified people do not review.
6.1 Reviewer interests
Our results suggest the lack of interest is one of the most common problems in paper-reviewer matching, for both authors and reviewers. The authors are aware of this problem and sometimes try to optimize for it by pursuing the "safe", popular topics. Unenthusiastic reviewers will likely produce shallow, heuristic-based reviews, essentially penalizing non-mainstream research. Both tendencies contribute to ossification of the field (Chu and Evans, 2021), and generally need to be minimized.
It is in the AC's interest to find interested reviewers, since that minimizes late reviews, but they need to know who finds what interesting. That is not as simple as a match by topic/methodology, clear from the publication record. Interests change not only gradually over time but also according to what is popular or salient at the given moment (Tversky and Kahneman, 1974; Dai et al., 2020), or even in seemingly irrational ways (e.g. by being sensitive to the framing of the problem) (Tversky and Kahneman, 1981). But although experience and knowledge may provide more stable descriptions of a reviewer, looking into dated publication records may be fundamentally counter-productive. According to one of our respondents: "I prefer the conferences who offer bidding processes to select the papers to review... I am more enthusiastic to review the papers compared to conferences that
assign papers based on what my interests were x years ago."
Bidding however has its own set of problems, including the practical impossibility to elicit all preferences over a big set of papers, the possibility of collusion rings (Littman, 2021), and, as one of our respondents put it, "biases towards/against certain paper types when bidding is enabled". But these problems potentially have solutions: there is work on detecting collusion rings (Boehmer et al., 2022), and several respondents suggested that bidding could be facilitated by subsampling with either keyword- or affinity-score-based approaches.
We support some of our respondents' recommendation for a combination of interest-based and non-interest-based (within a matching area) assignments, with the latter clearly marked as such for ACs and reviewers, and separate playbooks for the two cases. The reviewer training programs should aim to develop the expectation that peer review is something that combines utility and exploration.
We readily acknowledge that, like with any surveys with voluntary participation, our sample of respondents may not be representative of the field overall, since the people who have had issues with peer review system are more incentivized to respond. However, precisely for that reason this methodology can be used to learn about the commonly reported types of problems, which was our goal. Our response rate turned out to be comparable to the response rate of the official ACL survey soliciting feedback on its peer review reform proposal (Neubig, 2020), which received 199 responses.
It is an open problem how future conferences could systematically improve, if they cannot rely on surveys to at least reliably estimate at what scale an issue occurs. Asking about satisfaction with reviews does not seem to produce reliable results (Daumé III, 2015; Cardie et al.). Our survey included a question about satisfaction with the paper-reviewer matching, and whether the most recent experience was better or worse than on average. Both reviewers and authors were more satisfied than dissatisfied, and considered the recent experience better than on average, despite reporting so many issues (see App. fig. 11 for the distribution).
6.3 Interpretable Paper-Reviewer Matching: Problem Formulation
There already are many proposed solutions for paper-reviewer matching (see section 2), but their evaluation is the more difficult problem. The obvious approach would be to use bidding information or real assignments made by ACs as ground truth, but this data is typically not shared to protect reviewer anonymity. It would also provide a very noisy signal not just due to different assignment strategies between ACs, but also different quality of assignments depending on how much time they have on a given day. Both ACs and bidding reviewers are also likely<sup>8</sup> to favor top-listed candidates. And, as our findings suggest, the optimal assignment strategies in an interdisciplinary field might genuinely vary between different types of papers and tracks. A system unaware of that might systematically disadvantage whole research agendas.
Given that even the human experts cannot tell what the best possible assignments are, we propose to reformulate the problem as interpretable paper-reviewer matching. That problem is not the same as the problem of faithfully explaining why a given paper-reviewer matching system produced a certain score, for which we have numerous interpretability techniques (Søgaard, 2021). The AC goal is fundamentally different: not to understand the system, but to quickly find the information that the  $\mathrm{AC^9}$  considers relevant for making the best possible match. Therefore the task of interpretable paper-reviewer matching is rather to help to identify the information that the stakeholders wish the decisions to be based on, and to provide that information as justification for the decisions.
We present the results of the first survey on paper-reviewer assignment from the perspective of three groups of stakeholders in the NLP community: authors, reviewers, and ACs. The results point at a host of issues, some immediately actionable (e.g. providing the ACs with better information), some normative (e.g. different kinds of research may need different assignment strategies), and some open (e.g. how do we evaluate the effect of any changes to peer review process?) A big issue for
both authors and reviewers is mismatches due to lack of interest, which is in tension with explorative aspects of peer review. We recommend to address this issue with a combination of assignments based on bidding and random matches within area, backed up by reviewer training.
Many thanks to Marzena Karpinska, Friedolin Merhout, and the anonymous reviewers for their insightful comments. We would also like to thank all our survey respondents, without whom this study would not have been possible.
Broader impact. The study identifies types of information that could be used to provide better paper-reviewer matches. Used strategically by a conference, it could be a powerful tool for diversifying the field, by helping the non-mainstream papers find the reviewers more open to them. By the same token, if the entity organizing the review process aimed for suppressing such research, deprioritising this information could harm such papers. Our proposal of interpretable paper-reviewer assignments would mitigate this potential risk by requiring the organizers to disclose their rationale for any given match.
Personal data. The surveys are designed to not solicit any personally identifiable information (including comments about individual peer review cases in the past conferences), or demographic information about participants.
Potential risks. The respondents are participants in anonymous peer review process, and as such being tracked back to individual peer review cases could expose them to retaliation. The survey therefore did not solicit information about specific venues (only broader categories such as "  $\ast$  ACL conferences"), and we manually verified that the open-ended comments also do not contain references to specific cases. We thus foresee no potential risks from deanonymization of the respondents.
Informed consent. The respondents are informed about the organizers and the objective of the study: to identity current practices of paper-reviewer assignments in CL/NLP conferences and ways in which this process can be improved. Responses are anonymous and respondents consent
<sup>8</sup>Position bias is well documented in search & recommendation systems (Craswell et al., 2008; Collins et al., 2018). <sup>9</sup>Or the program chairs, should the conference aim to have consistent policies for all ACs.
to the use and sharing of their responses for research purposes. Respondents must give consent to continue the survey.
Intended use. The survey data and forms will be made publicly available for research purposes.
Institutional approval. The study was approved by the Research Ethics Committee at the authors' institution.
Balazs Aczel, Barnabas Szaszki, and Alex O. Holcombe. 2021. A billion-dollar donation: Estimating the cost of researchers' time spent on peer review. Research Integrity and Peer Review, 6(1):14.
Emily M. Bender. 2019. The #BenderRule: On Naming the Languages We Study and Why It Matters.
Emily M. Bender and Leon Derczynski. 2018. Paper Types.
Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. 2021. The NeurIPS 2021 Consistency Experiment.
Niclas Boehmer, Robert Bredereck, and André Nichterlein. 2022. Combating Collusion Rings is Hard but Possible.
Claire Cardie, Iryna Gurevych, and Yusuke Miyao. ACL 2018: A Report on the Review Process of ACL 2018.
Haw-Shiuan Chang and Andrew McCallum. 2021. Cold-start paper recommendation using multi-facet embedding.
Johan S. G. Chu and James A. Evans. 2021. Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences, 118(41).
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
Andrew Collins, Dominika Tkaczyk, Akiko Aizawa, and Joeran Beel. 2018. A Study of Position Bias in Digital Library Recommender Systems. arXiv:1802.06565 [cs].
Corinna Cortes and Neil D. Lawrence. 2021. Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv:2109.09774 [cs].
Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, pages 87-94, New York, NY, USA. Association for Computing Machinery.
Jane Dai, Jeremy Cone, and Jeff Moher. 2020. Perceptual salience influences food choices independently of health and taste preferences. Cognitive research: principles and implications, 5(1):2-13.
Hal Daumé III. 2015. Some NAACL 2013 statistics on author response, review quality, etc.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Richard Gordon and Bryan J. Poulin. 2009. Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant. Accountability in Research, 16(1):13-40.
David Hartvigsen, Jerry C. Wei, and Richard Czuchlewski. 1999. The Conference Paper-Reviewer Assignment Problem*. Decision Sciences, 30(3):865-876.
Laura Hyman, Julie Lamb, and Martin Bulmer. 2006. The use of pre-existing survey questions: Implications for data quality. In Proceedings of the European Conference on Quality in Survey Statistics, pages 1-8. Cardiff Wales, UK.
Xinlian Li and Toyohide Watanabe. 2013. Automatic Paper-to-reviewer Assignment, based on the Matching Degree of the Reviewers. Procedia Computer Science, 22:633-642.
Michael L. Littman. 2021. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43-44.
Mingnan Liu and Laura Wronski. 2018. Trap questions in online surveys: Results from three web survey experiments. International Journal of Market Research, 60(1):32-49.
Cheng Long, Raymond Chi-Wing Wong, Yu Peng, and Liangliang Ye. 2013. On Good and Fair Paper-Reviewer Assignment. In 2013 IEEE 13th International Conference on Data Mining, pages 1145-1150.
Graham Neubig. 2020. ACL Rolling Review Proposal.
Graham Neubig, John Wieting, Arya McCarthy, Amanda Stent, Natalie Schluter, and Trevor Cohn. 2021. ACL Reviewer Matching Code. Association for Computational Linguistics.
OpenReview. 2021. Paper-reviewer affinity modeling for OpenReview.
Eric Price. 2014. The NIPS experiment.
Priscilla Rasmussen. 2021. Acl membership report 2012-2021 summer statistics. ACL Admin Wiki.
Anna Rogers and Isabelle Augenstein. 2020. What Can We Do to Improve Peer Review in NLP? In Findings of EMNLP, pages 1256-1262, Online. Association for Computational Linguistics.
Nihar B Shah. 2022. An overview of challenges, experiments, and computational solutions in peer review (extended version). Communications of the ACM.
Anders Søgaard. 2021. Explanable Natural Language Processing. Synthesis Lectures on Human Language Technologies, 14(3):1-123.
Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. 2019. PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review. In Proceedings of the 30th International Conference on Algorithmic Learning Theory, pages 828-856.
Seymour Sudman and Norman M. Bradburn. 1973. Effects of time and memory factors on response in surveys. Journal of the American Statistical Association, 68(344):805-815.
A Tversky and D Kahneman. 1981. The framing of decisions and the psychology of choice. Science (American Association for the Advancement of Science), 211(4481):453-458.
Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131.
Fan Wang, Ning Shi, and Ben Chen. 2010. A comprehensive survey of the reviewer assignment problem. International Journal of Information Technology & Decision Making, 09(04):645-668.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Simple and effective paraphrastic similarity from parallel translations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4602-4608, Florence, Italy. Association for Computational Linguistics.
Fei Xia, Wenjie Li, and Roberto Navigli. 2021. ACLIJCNLP 2021 Program Chair Report.
H. Öztas Ayhan and Semih Isiksal. 2005. Memory recall errors in retrospective surveys: A reverse record check study. *Quality and Quantity*, 38(5):475-493. Copyright - Kluwer Academic Publishers 2004; Last updated - 2021-09-09.
In this appendix we introduce supplementary figures and tables.
Figure 8: We ask reviewers and authors whether they know for certain, or maybe knows, or do not know how the paper-reviewer matching was done for their last CL/NLP venue. We then ask both reviewers, authors and ACs what they believe (or knows in the case of some) was the process for this venue The guesses, and knowledge herof, are much different from best options in fig. 4, discussed in section 5.
Figure 9: Distribution of non-students in the three roles, with overlap derived from asking the question Did you also serve as a reviewer/author? for their last CL/NLP venue.
(a) AC/Action Editors
(b) Reviewers
Figure 10: The boxplots show the number of assignments given and optimal for a) ACs and b) Reviewers, discussed in section 5.3. Number of given assignments are reported for the venue in which the respondent last served as AC/reviewer, and optimal number of assignments are reported for time periods one week for ACs and one month for reviewers. Mean given and optimal number of assignments, across all respondents/venues, are shown with vertical striped lines.
Figure 11: We ask all respondents general questions about their satisfaction with their last experience as an AC/reviewer/author and their overall satisfaction in this role. We discuss these results in the limitations section 6.2.
(b) Question: would you say your most recent experience with paper-reviewer matching/paper assignment(s)/set of reviews described above, was better or worse than on average?
AREA CHAIR	REVIEWER	AUTHOR
1-5	6-10	>10	1-5	6-10	>10	1-5	6-10	>10
Bachelor's student	0	0	0	0	0	0	0	1	0
Master's student	0	0	0	1	0	0	0	0	0
PhD student	0	1	0	7	10	3	25	6	2
Other	0	0	0	0	0	1	2	0	1
Postdoc	0	1	2	1	3	12	2	0	11
Academic researcher	0	0	19	4	1	21	4	0	13
Industry researcher	1	0	6	5	0	8	1	2	3
Total	1	2	27	18	14	45	34	9	30
Table 1: This table shows the count of respondents from each role (AC/reviewer/author) reporting one of 7 career statuses and an amount of times having had their own papers reviewed. The numbers reflects those plotted in fig. 2, section 4.1.
Tasks	Bidding	Method	Type of paper	Trust	Data	Affinity Score
AC / Action Editor	3.90±0.83	3.67±1.25	3.70±0.74	3.37±0.95	3.27±1.67	2.83±1.00	2.13±1.50
Reviewer	4.00±0.90	4.03±1.07	3.86±0.85	3.16±0.97	2.96±1.57	2.75±1.09	1.97±1.38
Author	3.95±0.86	3.86±1.22	3.59±0.87	3.45±1.18	3.11±1.60	2.84±0.98	1.85±1.32
Grand mean	3.95	3.85	3.72	3.32	3.11	2.81	1.99
Table 2: Mean importance with  $0 = \text{Not sure}$ ,  $1 = \text{Not important}$ ,  $2 = \text{Slightly important}$ ,  $3 = \text{Moderately important}$ ,  $4 = \text{Very important}$  and  $5 = \text{Extremely important}$ , for the seven paper-reviewer matching factors shown in fig. 7. Removing "Not sure" does not change the overall ranking. The grand mean is the unweighted mean of ACs', reviewers' and authors' mean scores. The mean absolute difference is greatest between ACs and reviewers (0.20) and smallest between ACs and authors (0.12), while between reviewers and authors it is 0.16. These results are discussed in section 5.2 under "Stakeholder preferences".
Tasks	Bidding	Method	Type of Paper	Trust	Data	Affinity Score
Computationally-aided linguistic analysis	1.000	0.624	0.205	0.089	0.699	0.035<	0.654
NLP engineering experiment paper	1.000	0.716	0.038>	0.135	0.270	0.772	0.699
Reproduction paper	0.457	0.766	0.055	0.601	1.000	0.004>	0.562
Resource paper	0.728	0.433	0.727	0.162	0.818	1.000	0.616
Position paper	0.222	0.766	0.433	0.135	0.236	0.493	0.003<
Survey paper	1.000	0.186	0.738	0.420	1.000	0.840	0.480
Other	0.601	0.052	1.000	0.019<	0.733	0.202	0.063
Table 3: P-values of two-sided Fisher Exact tests, discussed in section 5.2. For each contribution type, we test the null hypothesis that there is no difference in whether respondents find a paper-match factor (from fig. 7) more than or less than moderately important, depending on whether or not individuals have worked on the specific types of papers (contribution types). For each contribution type  $i$  and paper-match factor  $j$ , a  $2 \times 2$  contingency table is made with the counts of a) respondents having worked with type  $i$  and finding factor  $j$  less than moderately important, b) having worked with type  $i$  and finding factor  $j$  more than moderately important, c) having not worked with  $i$  and finding  $j$  less than moderately important, d) having not worked with  $i$  and finding  $j$  more than moderately important. The p-values reflect the probability of observing the given counts or something more imbalanced between those having and not having worked on type  $i$ . Significant p-values,  $p < 0.05$ , are in bold, and for these, superscript  $>$  denotes that respondents having worked with  $i$  believe factor  $j$  is more than moderately important, and the superscript  $<$  denotes the opposite.