SlowGuess/NAACL_2022_OCR huggingface dataset:
Terne Sasha Thorn Jakobsen<sup>1</sup>, Anna Rogers<sup>1,2</sup>
$^{1}$  Copenhagen Center for Social Data Science, University of Copenhagen
2 RIKEN Center for Computational Science
1terne.thorn@sodas.ku.dk, 2arogers@sodas.ku.dk
Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.
Peer review is increasingly coming under criticism for its arbitrariness. Two NeurIPS experiments (Price, 2014; Cortes and Lawrence, 2021; Beygelzimer et al., 2021) have shown that the reviewers are good at identifying papers that are clearly bad, but the agreement on the "good" papers appears to be close to random. Among the likely reasons for that are cognitive and social biases of NLP reviewers (see overview by Rogers and Augenstein, 2020), fundamental disagreements in such an interdisciplinary field as NLP, and acceptance rates that are kept low<sup>1</sup> irrespective of the ratio of high-quality submissions.
Such arbitrariness leads to understandable frustration on the part of the authors whose jobs and graduation depend on publications, and it also means lost time and opportunities (Aczel et al., 2021; Gordon and Poulin, 2009) for science overall. Reviews written by someone who does not have the requisite expertise, or does not even consider the given type of research as a contribution, it is a
Figure 1: Overview of all respondents and overlap of their roles for their last experience at NLP venues.
loss for all parties: the authors do not get the intellectual exchange that could improve their projects and ideas, and reviewers simply lose valuable time without learning something they could use. It is also a loss for the field overall: less popular topics could be systematically disadvantaged, leading to ossification of the field (Chu and Evans, 2021).
This paper contributes a snapshot of this problem in NLP venues, based on a survey of authors, reviewers and area chairs (ACs). We collected 180 responses, which is is comparable to the volume of feedback collected for implementing the ACL Rolling Review (ARR). The overall distribution of respondents' roles is shown in fig. 1. We present the commonly reported issues and community preferences for different paper assignment workflows (section 4). We derive actionable recommendations to how peer review in NLP could be improved (section 5), discuss the limitations of survey methodology (section 6.2), and conclude with desiderata for interpretable peer review assignments (section 6.3).
2 Background: Peer Review in NLP
Paper-reviewer assignments are matches between submissions to conferences or journals and their available pool of reviewers, taking into account the potential conflicts of interest (COI) and reviewer assignment quotas.
$^{1}$ https://twitter.com/tomgoldsteincs/status/1388156022112624644
Human Language Technologies, pages 4810 - 4823
Among the systems used in recent NLP conferences, the Softconf matching algorithm takes into account bidding, quotas, and manual assignments, and randomly assigns the remaining papers as evenly as possible<sup>2</sup>. NAACL and ACL 2021 used SoftConf, but also provided their ACs with affinity scores produced by a "paraphrastic similarity" system based on an LSTM encoder, which is trained on Semantic Scholar abstracts (Wieting et al., 2019; Neubig et al., 2021). Affinity scores are scores indicating how well a given submission matches a given reviewer. They are typically computed as the similarity (e.g. cosine similarity) between the embeddings of certain information about the submission and the reviewer's publication history (e.g. abstracts and titles).
ARR switched to OpenReview and currently uses their SPECTER-MFR system (OpenReview, 2021) which is based on SPECTER (Cohan et al., 2020) and MFR embeddings (Chang and McCallum, 2021) for computing affinity scores. The assignments are then made with the MinMax matching algorithm<sup>4</sup>.
The problem of paper-reviewer assignment is by itself an active area of research (see overview of key issues for CS conferences by Shah (2022)). There are many proposals for paper-reviewer assignment systems (Hartvigsen et al., 1999; Wang et al., 2010; Li and Watanabe, 2013, inter alia), some of which also consider the problem of "fair" assignments (Long et al., 2013; Stelmakh et al., 2019). Such studies tend to be hypothesis-driven: they make an assumption about what criteria should be taken into account, design a system and evaluate it. To the best of our knowledge, ours is the first study in the field to address the opposite question: what criteria should be taken into account, given the diversity of perspectives in an interdisciplinary field? We take that question to the community.
3 Methodology: survey structure and distribution
We developed three separate surveys for the main groups of stakeholders in the peer review process: authors, reviewers and ACs.
They follow the same basic structure: consent to participation (see Impact Statement), background
information, questions on most recent experiences in the role which the survey pertains to, and how the respondents believe paper-reviewer matching should be performed. Most questions are asked to respondents in all three roles, reformulated to match their different perspectives.
The responses were collected late 2021 and all respondents are required to confirm that their most recent experience as an AC/reviewer/author is in 2019-2021. The full surveys and response data are publicly available<sup>5</sup>.
Participant background. All surveys include questions on career status and the number of times the respondents have been ACs/reviewers/authors at NLP venues. We ask what venues they have experience with (as broad categories) and what types of contributions they make in their work.
Participant experience with peer review. We further ask the respondents a range of questions about their experience as AC/reviewer/author: how satisfied they are with the process, what issues they have experienced, what was the assignment load (ACs and reviewers), how paper-reviewer matching was done, how they would prefer it to be done, and which factors they believe to be important for paper-review matching. Most of the questions are multiple-choice, with addition of some open-ended questions where appropriate, so that respondents can elaborate their answers or add to the available options. Whenever possible, the question formulations were taken from the question bank of UK Data Service (Hyman et al., 2006). Attitude questions use a 5-point Likert scale.
Limited memory is an important concern in surveys (Sudman and Bradburn, 1973; Oztas Ayhan and Isiksal, 2005), and we cannot expect the respondents to accurately recall all their experience with peer review. To reduce memory recall errors, the survey focuses on the respondent's most recent experience, but they also have a chance to reflect on prior experience in open-ended questions, and to report whether they experienced certain issues at any time in their career.
Survey distribution. We distributed the surveys via three channels: by handing out flyers at EMNLP 2021, through mailing lists (ML-news, corpora list, Linguist list), and through Twitter with the hashtag #NLProc. Participation was voluntary,
$^{5}$ https://github.com/terne/Paper-Reviewer-Matching-Surveys
$^{2}$ https://www.softconf.com/about/index. php/start/administration-view  
 $^{3}$ Source: personal communication with the ARR team.  
 $^{4}$ https://github.com/openreview/openreview-matcher
Figure 2: Career status of the respondents vs their experience receiving peer review. Numerical data is available in table 1 in the appendix.
with no incentives beyond potential utility of this study for improving NLP peer review.
Data validation. Given that links to surveys were distributed openly and that we did not ask for any identifiable information, the surveys needed to include other means of validation to ensure that the responses included in the analysis were from attentive, relevant individuals. Our approach for validating the data quality follows satisficing theory (Liu and Wronski, 2018), with the main safeguards being 1) the checking of response consistency, including a few "traps" where inconsistency or illogical responses can be exposed, and 2) the inclusion of open-ended questions.
$73\%$  ACs,  $40\%$  reviewers,  $33\%$  authors have provided at least one response to our open-ended questions, and we did not find any meaningless or incoherent comments not addressing the question. For consistency checks, all respondents were asked:
- How many times they have been an AC/reviewer/author. One of the options was "0", contradicting the earlier confirmation of experience in a given role.
- When was the last time they were an AC/reviewer/author. One of the options was "earlier than 2019", contradicting the earlier confirmation of peer review experience in 2019-2021.
- Whether they have performed the other roles. New authors may have not reviewed or AC-ed, but reviewers should also have been authors, and ACs should have experience with all roles.
Overall we received 38 responses from ACs, 87 from reviewers and 81 from authors (206 in total).
After removing 20 incomplete responses and 8 responses inconsistent with the "trap" questions, we report the results for 30 responses from ACs, 77 from reviewers and 73 from authors (180 in total).
4.1 Who are the respondents?
According to the past conference statistics, we could expect that many submissions would be primarily authored by the students, and reviewers are generally expected to be relatively senior, which should correspond to their going through peer review more often. We can use this expected pattern as an extra validation step for the survey responses.
Figure 2 shows that the responses are in line with this expected pattern. We received the most responses from academic researchers (62), PhD students (54), and postdocs (32). Most academic researchers and postdocs, but not PhD students, have had their work reviewed more than 10 times. At the same time  $65\%$  of the PhD students who served as reviewers went through peer review more than 5 times, as opposed to  $24.2\%$  of PhD students in the author role. Fewer industry than academic researchers responded to the survey. This could be related to the fact that a large part of the "academic" demographic are students – and in 2020-2021 the ACL membership among students was equal to or exceeding other demographics (Rasmussen, 2021).
4.2 Paper types
The next question is to see what kinds of research papers the respondents to our surveys have authored: engineering experiment, survey, position paper etc., according to the COLING taxonomy by Bender and Derczynski (2018). We expect that more senior researchers will have more experience
Figure 3: Types of research performed by respondents (multiple options could be selected).
with different types of work. Indeed, on average the authors have worked with 2.5 types of papers, vs. 3.0 for reviewers and 3.6 for ACs. The distribution is shown in fig. 3. The most respondents have authored engineering experiment papers (with the authors reporting the most such work).
Note that this only indicates whether the respondents to our surveys have or have not authored certain types of papers, rather than how many. In terms of volume, the engineering papers are a lot more prevalent: e.g. at ACL 2021 the "Machine learning" track had 332 submissions, vs 168 in the "Resources and evaluation" track (Xia et al., 2021).
4.3 What kinds of problems do people report?
As with any voluntary feedback, our surveys were likely to receive more responses from people who had a grievance with the current process. Indeed, we find that only  $6.7\%$  of ACs,  $20.5\%$  of authors, and  $22.1\%$  of reviewers say that they have not had any issues in their last encounter with NLP venues.
The overall distribution for the types of problems reported by the authors, reviewers and ACs in their last and overall experience is shown in fig. 5. Given that at the time of this survey the ARR was recently deployed as the only ACL submission channel, we highlight the responses from the people for whom the most recent venue was ARR:  $28\%$  reviewers,  $18\%$  authors,  $50\%$  ACs.
- Two of the most frequent complaints of ACs (about  $50\%$  of the respondents) are insufficient information about reviewers and clunky interfaces;
- Many paper-reviewer mismatches (about  $30\%$ , if the report of the last experience is representative) are avoidable: they should have been clear from the reviewers' publication history;
Over a third of the author respondents in their last submission (about  $50\%$  over all history) received reviews from reviewers lacking either expertise or interest, and that is supported by the reviewers' reports of being assigned papers that were mismatched on one of these dimensions;
- The authors report that many reviews (over a third in last submission, close to  $50\%$  overtime) are biased or shallow, which might be related to the above mismatches in expertise or interest;
- Two patterns are exclusive to ARR: insufficient time for  $\mathrm{ACs}^6$ , and zero authors with no issues.
Figure 4: Which of the following options would you consider best for assigning reviewers to submissions?
Topics mentioned in the open-ended comments
(See supplementary materials for full categorized comments)
ACs: bidding (2), similarity+manual (1), similarity+bidding+manual (5), keyword-based filtering + bidding (2), similarity (1), tracks (1), other info (2), ARR (1), interface (1)
Reviewers: manual (2), similarity + bidding (3), similarity+bidding+manual (3),keywords (1),keywords+similarity (1),tracks (2), tracks+bidding (1), other (4)
Authors: against similarity (2), similarity + bidding (2), similarity+bidding+manual (2),ARR (2), random (2)
4.4 Knowledge of the workflow
Our next question is what methods NLP venues use to match submissions to reviewers, and to what extent the stakeholders (authors and reviewers) are aware of how it is done. We find that relatively few authors  $(23.3\%)$  and reviewers  $(23.4\%)$  know for sure what process was used, which begs for more transparency in the conference process. The ACs report that the most frequent case  $(37\%)$  is a combination of automated and manual assignments. Interestingly, most reviewers believe that their assignments were automated  $(36\%)$ , and only  $(28\%)$  believe they were automated+manual. See App. Figure 8 for full distribution.
5 The Ideal Process
5.1 Ideal workflow
When asked about what paper-assignment process they would prefer (given that fully manual matching is impractical for large conferences), most ACs and authors opted for automated+manual process, but for the reviewers this is the second preferred process  $(26\%)$  with  $30\%$  opting for bidding  $^+$  manual checks (see fig. 4). There was also relatively
$^{6}$ ARR has since switched to 6-week cycles, which might help to address this issue (https://aclrollingreview.org/six-week-cycles/).
Figure 5: The issues with peer review process, reported by ACs, reviewers and authors, in their last (on the left) versus historical (on the right) experience with CL/NLP venues.
(The full comments categorized by these topics can be found in the survey data repository)
Area chairs: interface issues (7), bad reviewers/reviews (5), workload issues (6), issues with ARR (4), lacking information on reviewers (4), communication issues between both systems and other human agents (4), lack of qualified reviewers in the pool (3), issues with meta-reviews (2), affinity score complaints (2), affinity score for finding reviewers the AC does not know personally (1), preference for manually recruited reviewers (1), papers assigned to ACs outside their area of expertise (1), too many declines (1), mismatch in goals of reviewers and authors (1), emergency reviews (1), bidding enabling bias (1).
Reviewers: choices forced by ACs (5), preference for bidding (4), areas of past expertise not currently of interest (4), lack of interest in the paper (3), methodological mismatch between generations of NLP researchers (3), mismatch in research methods (2), publication records as an unreliable indicator for assignments (1), mismatch in languages (1), time issues (1), reviewer bias (1)
Authors: reviewer expectation for a certain kind of research (6), inattentive reviews (5), short reviews (3), mismatch between the score and the text of the review (3), requests for irrelevant citations (2), confirmation bias (1), non-constructive criticism (1), shallow reviews (1), lack of reviewer competence (2), missing reviews (2), requests for irrelevant comparisons (1), “wild” estimates of impact (1), unannounced policy changes (1)
large support for pure bidding (13-18% of respondents in all roles), and cumulatively pure bidding and bidding with manual adjustments have as much or more support from all respondent categories than the automated matching + manual assignments.
The analysis of open-ended comments suggested that the respondents were aware that bidding is quite labor-intensive on the part of the reviewers. 5 ACs, 3 reviewers and 2 authors suggested using affinity scores to filter the papers on which bids
would be requested, followed with manual checking. Another suggestion was keywords or more fine-grained areas/tracks, potentially as alternative to affinity scores for filtering down the list of papers to bid on. One AC suggested "an extensive, but still finite, set of tags (e.g. an ACL-version of ACM CCS concepts, or FAccT's submission tags)". One reviewer stressed that the keywords should be provided by the authors, to match what they perceive to be the salient aspects of the paper.
1 reviewer and 1 author suggested looking at whether the paper cites the potential reviewer<sup>7</sup>, as this could be a good indicator for the reviewer's interest. 1 reviewer and 2 authors voiced support for some randomness in the assignments (given a track-level match): "Bidding + some random assignment to ensure diversity in the matching. We don't want reviewers to review only papers they *want* to review. However these random assignments should be clearly indicated to all, and treated accordingly."
5.2 Ideal assignment criteria
AC past experience. Figure 5 shows that one of the most common problems for the ACs is that they were not provided with enough information to facilitate the paper-reviewer matching. The follow-up question is what information they are provided with, and how useful they find it.
Figure 6 shows that the types of information with the highest utility information are links to reviewer profiles, bidding information, and affinity scores. But affinity scores are also the most controversial: it is the type of information that the most ACs find "not very useful" or "not useful at all" (20%).
Overall the results suggest that ACs are presented with little structured information about reviewers, and have to identify the information they need from a glance at the reviewers' publication record. Seniority, expertise, and reviewer history notes from other ACs are all reported to be useful, but they were never provided directly to many ACs.
An avenue for future research is offered by three types of information that the most ACs are not sure about, presumably because they are rarely provided: structured information about the methods that the reviewers were familiar with, the languages they spoke, and affinity score explanations. We will show below that there is much support for taking such methods into account. For the languages, this might be due to the "default" status of English (Bender, 2019). We hypothesize that providing this information would make it easier to provide better matches for papers on other languages, which would in turn encourage the authors to submit more such work. Affinity will be discussed in section 6.3.
Stakeholder preferences. We then asked the respondents what factors they believe are important
for paper-reviewer assignments. Their answers are shown in fig. 7. The overall mean importance rankings (on scale 0-5) are as follows:
3.95 Reviewer has worked on the same task
3.85 Reviewer bid on the paper
3.72 Reviewer has worked with the same method
3.32 Reviewer has authored the same type of paper
3.11 AC knows & trusts the reviewer
2.81 Reviewer has worked with the same kind of data
1.99 The affinity score is high
The fact that affinity scores rank the least important for NLP researchers (who would know the most about them) is interesting, and perhaps related to the fact that evaluation of paper-reviewer matching systems remains an open problem, with little empirical evidence for how well our current systems really work. In the absence of such evidence, our results suggest that the respondents across all groups are not very positive about their experience with such systems. In the authors' personal experience, when the conference chairs provide automated affinity scores they caution the area chairs against fully relying on them, and urge to adjust the assignments manually.
Our data suggests that within groups of stakeholders the individual variation in importance of different factors is higher for some factors and stakeholders than others: e.g. ACs vary within 1 point on the importance of knowing the data, but only within 0.74 points on importance of knowing the tasks. This has implications for approaches who would rely on AC assignments as ground truth for automated assignment systems: they could end up modeling the annotator instead of the task (Geva et al., 2019). See App. table 2 for full data.
We then explored the question of whether the experience of having authored research of a certain type correlates with any changes in the attitude towards some of these paper-reviewer matching factors. For each pair of type of research and matching factor, we ran two-sided Fisher's Exact tests for all respondents who have authored (or not) the types of research and the importance they attached to different factors in paper-reviewer assignment (binning on less than moderately important and more than moderately important). For some pairs there were statistically significant differences: e.g. the respondents who have authored reproduction papers were significantly more likely to believe it important that the reviewer has worked with the same kind of data  $(p = 0.004)$ , and respondents who authored position papers were significantly less likely
<sup>7</sup>We believe this is an interesting idea, but it could lead to authors strategically placing citations to maximize the chances of acceptance, or being punished for citing work that they may criticize or claim to improve upon.
Figure 6: The diverging bars show the experienced utility of different kinds of information about reviewers that ACs may have been presented with to assist in manual checks of paper-reviewer matches. If the respondent had never been presented with the specific kind of information they chose "Never provided".
Figure 7: Question: How important do you think the following factors are for a good paper-reviewer match?
to believe a high automated affinity score is important  $(p = 0.003)$ . See table 3 in the appendix for all  $p$ -values and more details on the tests. We note that the relationships are not necessarily causal.