A Gold Standard Dataset for the Reviewer Assignment Problem
Authors: Ivan Stelmakh
John Frederick Wieting
Yang Xi
Graham Neubig
Nihar B Shah
Venue: Accepted by TMLR ()

Abstract: Many peer-review venues are using algorithms to assign submissions to reviewers. The crux of such automated approaches is the notion of the “similarity score’’ — a numerical estimate of the expertise of a reviewer in reviewing a paper — and many algorithms have been proposed to compute these scores. However, these algorithms have not been subjected to a principled comparison, making it difficult for stakeholders to choose the algorithm in an evidence-based manner. The key challenge in comparing existing algorithms and developing better algorithms is the lack of the publicly available gold-standard data that would be needed to perform reproducible research. We address this challenge by collecting a novel dataset of similarity scores that we release to the research community. Our dataset consists of 477 self-reported expertise scores provided by 58 researchers who evaluated their expertise in reviewing papers they have read previously. 

We use this data to compare several popular algorithms currently employed in computer science conferences and come up with recommendations for stakeholders. Our four main findings are:
- All algorithms make a non-trivial amount of error. For the task of ordering two papers in terms of their relevance for a reviewer, the error rates range from 12%-30% in easy cases to 36%-43% in hard cases, thereby highlighting the vital need for more research on the similarity-computation problem.
- Most specialized algorithms are designed to work with titles and abstracts of papers, and in this regime the Specter2 algorithm performs best.
- The classical TF-IDF algorithm which can use full texts of papers is on par with Specter2 that uses only titles and abstracts.
- The performance of off-the-shelf LLMs is worse than the specialized algorithms.
We encourage researchers to participate in our survey and contribute their data to the dataset here: https://forms.gle/SP1Rh8eivGz54xR37

Published in Transactions on Machine Learning Research (04/2025)
A Gold Standard Dataset for the Reviewer Assignment Prob- lem
Ivan Stelmakh, John Wieting, Sarina Xi, Graham Neubig, and Nihar B. Shah
Carnegie Mellon University
Reviewed on OpenReview:
Abstract
Many peer-review venues are either using or looking to use algorithms to assign submis- sions to reviewers. The of such automated approaches the notion of the "simålarity score" a numerical estimate of the expertise of reviewer in reviewing paper and many algorithms have been proposed to compute these scores. However, these algorithms have not been subjected to a principled comparison, making it difficult for stakeholders to choose the algorithm in an evidenco-based manner. The key challenge in comparing existing algo- rithms and developing better algorithuns the lack of the publicly available gold-stnndard data that would be needed perform reproducible research. We address this challenge by collecting novel dataset of similarity scores that release to the research community. Our dataset consists of 477 self-reported expertise scores provided by 58 researchers who evaluated their expertise in reviewing papers they have read previously.
We use this data to compare several popular algorithms currently employed in computer science conferences and come with recommendations for stakeholders. Our four main findings are:
All algorithms make a non-trivial amount of error. For the lask of ordering two papers in terms of their relevance for a reviewer, the error rates range from 12%-30% in easy cases to 36%-43% in hard cases, thereby highlighting the vital need for more research problem.
on the
Most specialized algorithms are designed to work with titles and abstracts of papers, and this regime the SPECTER2 algorithm performs best.
The classical TF-IDF algorithm which can use full texts of papers is on par with SPECTER2 that uses only titles and abstracts.
The performance of off-the-shelf LLMs worse than the specialized algorithms.
We encourage researchers to participate in survey and contribute their data to the dataset here
1 Introduction
Assigning papers to reviewers with appropriate expertise is the key prerequisite for high-quality reviews (Thurner & Hanel 2011 Black et al. 1998 Bianchi & Squazzoni 2015). Even a small fraction of incorrect
reviews can negatively impact the quality of the published scientific standard [Thunner Hanel 2011] l and
hurt carcers of rescarchers (Merton 1968 Squazzoni & Gandelli 2012 Thorngate & Chowdhury 2014
Quoting Triggle & Triggle 2007):
"An incompetent review may lead to the rejection of the submitted paper, or of the grant application, and the ultimate failure of the career of the author."
* *Corresponding author: cihaars

---

Published in Transactions on Machine Learning Research (04/2025)
Conventionally, the selection of reviewers for submission the task of a journal editor a program committee of conference. However, the rapid growth in the number of submissions to many publication venues (Sah 2022b) has made the manual selection of reviewers extremely challenging. As result, many
pecr-review venues in computer science as well as other fields are either using or looking to use algorithms
to assist organizers in assigning submissions to reviewers (Garg et al. 2010 Charlin & Zemel 2013 Kobren
ct al. 2019 Kerzendorf et al., 2020 Stelmakh et al. 2021 OpenReview 2022).
The key component of existing automated approaches for assigning reviewers to submissions is the "similarity score". For any reviewer-submissio pair, the similarity score is number that captures the expertise of the reviewer in reviewing the submission. The assignment process involves first computing the similarity score for each submission-revieeer pair, and then using these scores to assign the reviewers to submissions (Shah, 2022b, Section 3). The similarity scores are typically computed by matching the text of the submission with the profile (e.g.; past papers) of the reviewer, which the primary focus of this paper. Additionally, these scores may also be augmented by manually selected subject arcas or reviewer- reviever-provided manual preferences ("bids").
Several algorithms for computing similarity scores have been already proposed and used in practice (we review these algorithms in Sections 2 such algorithms in a principled manner. First, there is no publicly available gold standard data that can be used for algorithm development. Second, despite the existence of many similarity-computate the absence of the gold standard data prevents principled comparison of these algorithuss. As a result, three flagship machine learning conferences ICML, NeurIPS, and ACL-rely on three different similarity- computation algorithms, and the differences in performance of these algorithms are not well understood.
and 5). However, there are two key challenges in designing and using
algorithms,
In this work, we address the aforementioned challenges and collect a dalasel of reviewers' expertise that can facilitate the progress in the reviewer assignment problem. Specifically, we conduct a survey of computer science rescarchers whose experience level ranges from graduate students to senior professors. In the survey, we ask participants to report their expertise in reviewing computer science papers they read over the last year.
Contributions First, collect and release high-quality dataset of reviewers' expertise that can be used for training and/or evaluation of similarity-coompulatiion algorithms. The dataset can be found on the project's GitHub page:
We use the collected datasel to compare existing similarity-compulation algorithms in terms of their accuracy in predicting the similarities, and inform organizers in making a principled choice for their venue. Specifically, we observe that when all algorithms operate with titles and abstracts of papers, the most advanced SPECTER2 algorithm performs best. However, when the much simpler TPMS is additionally provided with full texts of papers, it achieves the same level of performance. Generic large language models do not perform as well. We also conduct additional studies pertaining to 'pooling' methods and the use of tbssessts versus full papers.
Overall, we observe that all current algorithms exhibit non-trivial amounts of error. When tasked to compare a pair of papers in terms of the expertise of a given reviewer, all algorithms make 12%-30% mistakes even when two papers are selected to have a large difference in reviewer's expertisc. On a more challenging task of comparing two high-expertise papers, the algorithms err with probability 36%-43%. This observation underscores the vital need to design better algorithms for matching reviewers to papers.
Key limitations and a call for participation: It is important to keep mind the following limitations. The first limitation is its size, in that comprises 477 data points contributed by 58 researchers. Second, there is a prominent bias in terms of geographical skew with 74% of participants from the US.
Our dataset is not set in stone and we encourage researchers to participate in our survey and contribute their data to the dataset. By collecting more samples, we enable more fine-grained comparisons and also improve the diversity of the dataset in terms of both population of participants and subject areas of papers. Any contributions herein will benefit the peer-review process. The survey is available at:
2

---

Published in Transactions on Machine Learning Research (04/2025)
and we will be updating the released version regularly.
2 Related literature
In this section, we discuss relevant past studies. We begin with an overview of works that report comparisons of different similarity-computatton algorithms. We then provide brief discussion of the commonly used procedure for computing similarity scores, which are ultimately utilized to assign reviewers to submissions. Finally, we conclude with a list of works that design algorithms to automate other aspects of reviewer assignment.
Evaluation of compuuaotyyoo algorithms The OpenReview platform (OpenReview 2022) uses an approach of predicting authorship proxy to measuring the quality of the similarity scores computed by any algorithm. Specifically, they consider papers authored by number of researchers, remove one of these papers from the corpus, and predict expertise of each rescarcher in reviewing the selected paper. The performance of an algorithm then is measured as a fraction of times the author of the selected paper is predicted to be among the top reviewers for this paper. This authorship proxy, however, may not be representative of the actual task of similarity computation as algorithms that accurately predict the authorship relationship (and hence do well according to this approach) are not guarantood to accurately estimate expertise in reviewing submissions authored by other researchers.
Mimno & McCallum (007)) collected a dataset with external expertise judgments. Specifically, they used 148 papers accepted to the NeurIPS 2006 conference and 364 reviewers from the NeurIPS 2005 conference and ask human annotators - independent established researchers- - to evaluate expertise for a selected subset of 650 (paper, reviewer) pairs. Similarly, Zhao et al. (2022) collected the set of papers and reviewers in the ICIP 2016 conference. They then asked independent researchers to provide their estimates of the match between various reviewer-papee pairs. While this approach results in a publicly available dataset, we note that external expertise judgments may also be noisy judges may have incomplete information about the expertise of reviewers.
Dumais& & Nielsen 1992]. 1, Rodriguez & Bollen 20088) and Anjum et al. 2019 obtain more accurate expertise
judgments by relying on self reports of expertise evaluations from reviewers. In more detail, Dumais & Nielsen 1992 and Rodriguez & Bollen (2008) rely on ez-ante bids preferences of reviewers in reviewing submissions contrast. [Anjum et al. (2019) rely on ex-post evaluations of expert ise made by reviewers after reviewing the submissions. These works construct datasets that can be used to evaluate algorithms: Dumais & Nielsen
made advance of reviewing.
1992) employ 117 papers and 15 reviewers, Rodriguez & Bollen (0008) employ 102 papers and 69 reviewers and Anjum et al. (2019) employ 20 papers and 33 reviewers. However, Rodriguez/ & Bolleen 2008) and Anjum et al. (2019) use sensitive data that cannot be released without compromising the privacy of reviewers. Furthermore, ex-ante evaluations of Dumais & Nielsen (1992) and Rodriguez & Bollen (2008) may not have a high accuracy as (i) bids may contaminate expertise judgments with willingness to review submissions, and (ii) bids are based on a very brief acquaintance with papers. On the other hand, ex-post data by Anjum et al. (2019) is collected for papers assigned to reviewers using specific similarity- computation algorithm. Thus, while collected evaluations have high precision, they may also have low recall if the employed
algorithm erroneously assigned low expertise scores to some (paper, reviewer) pairs as evaluations of exportise for such papers were not observed.
In this work, we collect a novel dataset of reviewer expertise that (i) can be released publicly, and (ii) contains accurate self-evaluations of expertise that are based on a deep understanding of the paper and are not biased algorithm.
towards any existing ammiarity-compputtii
Similarity scores in conferences In modern conferences, similarity scores are typically computed by combining two types of input:
Initial automated estimates. First, a similarity- computation algorithm used to compute initial esti- mates. Many algorithms have been proposed for this task (Minno & McCallum 2007 Rodriguez &
3

---

Published in Transactions on Machine Learning Research (04/2025)
Bolen 2008
Charlin & Zemel 2013 Liu et al. 2014 Tran al. 2017 Anjum al. 2019: Kerzendorf
et al. 2020 Review
2022] and we provide more details on several algorithms used in flagship
computer science conferences in Section 5
Human corrections. Second, automated estimates are corrected by reviewers who can read abstracts of submissions and report bids prrseernnessss in reviewing the submissions. A number of works focus
on the bidding stage and (i) explore the optimal strategy to assist reviewers in navigating the pool of thousands of submissions (Fiez al.) 2020; Meir el. al. 2020) or (ii) protect the system from stralegic
bids made by colluding reviewers willing to get assigned to each other's paper (Jecmen et al. 2020 Wu
et al. 2021 Boehmer et al., 2021 Jecmen et al.
2022
Combining these two types of imput in a principled manner a non-trivial task. As result, different conferences use different strategies (Shah et al., 2018 Leyton-Brown et al., 2022 and there is lack of empirical or theoretical evidence that would guide venue organizers in their decisions.
Automation of the assigmment stage At a high level, automated assignment stage consists of two steps: first, similarity scores are computed: second, reviewers are allocated to submissions such that some notion of assignment quality (formulated terms the similarity scores) is maximized. In this work, we focus on the first stop of the process. Howover, for completeness, we now mention several works that design algorithms for the second step.
A popular notion of assignment quality is the cumulative similarity score, that is, the sum of the similarity scores across all assigned reviewers and papers. An algorithm pursuing such an objective is implemented in the widely employed TPMS assignment algorithm (Charlin & Zemel 2013) and similar ideas are explored in many papers (Goldsmith & Sloan 2007; Tang et al. 2010 Long et al. 2013 While the cumulative objective is a natural choice, has been noted that may discriminate against certain submissions by allocating all irrelevant reviewers to a subset of submissions, even when a more balanced assignment exists (Garg et al, 2010). Thus, number of works has explored the idea of assignment fairness, aiming at producing
balanced assignments (Kobren et al. 2019 Stelmakh et al.) 2021). Finally, other works explore the ideas of envy-freeness [Tan et al. 2021 Pay Payan 2022), resistance to lone-wolf strategic behavior (Xu et al. 2019
Dhull et al. 2022 and encouraging various types of diversity (Li et al. 2015 Leyton-Brown et al. 2022
3 Data collection pipeline
In this section, we describe the process of data collection. This work was performed under the approval of an Institutional Review Board (IRB).
Gold standard data In this study, we aim to collect a datasct of self-ovaluations of revicwers' expertise that satisfies two desiderata:
(D1) The dataset should comprise evaluations of expertise for papers participants read to a reasonable extent.
(D2) The dataset should be released publicly without disclosing any sensitive information.
(D3) We want the datasel to comprise experts' ratings of their own expertise rather than people judging other peoples' expertise.
Now, there are few natural approaches one could consider to collect dataset. For instance, one could consider data from the bidding process which may be publicly released (or at least subset of Shah 2022a but this will violate desideratum 1. An approach to satisfy desideratum 1 would be to observe that reviewers do read assigned papers at a conference and then release that data; however, one cannot release information about who reviewed which papers and hence this approach cannot satisfy desideratum 2. In order to satisfy both and 2, we could ask researchers to provide their opinions on the goodness of match between a set of papers and potential reviewers (Mimno & McCallum 2007), but this will violate desideratum 3 and may also violate desideratum 1.
4

---

Published in Transactions on Machine Learning Research (04/2025)
We thus took an approach in this paper that satisfies all three desiderata: asking rescarchers to provide their expertise levels for papers they have read. More specifically, researchers are asked to provide a list of papers they have read and provide a aating/rankingg of these papers according to their own expertise for reviewing them. We now discuss our data collection procodure in more detail.
Participant recruiting We recruited participants using combination of several channels that are typi- cally employed to recruit reviewers for computer science conferences:
Mailing lists. First, we sent recruiting emails to relevant mailing lists of several universities and research departments of companies.
Social media. Second, via a call for participation on X (formerly Twitter).
Personal communication. Third, we sent personal invites to rescarchers from our professional network.
We had screening criterion requiring that prospective participants have at least one paper published in the broad area of computer science. Overall, for the version of the dataset we release in this paper, we managed to recruit 58 participants, all of whom passed the screening.
Expertise evaluations The key idea our approach to expertise evaluation is to ask participants to evaluate their expertise in reviewingg popers they read in the past. After reading a paper, a rescarcher is in the best possible position to evaluate whether they have the right background- in terms of the techniques used in the paper and in terms of the broader research of the paper- to judge the quality of the paper. With this motivation, we asked participants to:
Recall 5-10 papers in their broud research area that they read to a reasomable extent in the last year and tell us their expertise in reviewing these popers.
In more detail, the choice of papers was constrained by two minor conditions:
The papers reported by participant should not be authored by them.
The papers reported by participant should be freely available online.
In addition to these constraints, we gave several recommendations to the participants in order to make the dataset more diverse and useful for the research purposes:
First, we asked participants to choose papers that cover the whole spectrum of their expertise with some papers being wellsseparated (c.g., very high expertise and very low expertise) and some papers being nearly-tied (e.g., two high-expertise papers).
Second, we recommended participants to avoid ties in their evaluations. To help participants comply with this recommendation, we implemented evaluation on a scale of 1 ("I am not qualified to review this paper") to 5 ("1 have background necessary to evaluate all the aspects of the paper") with a 0.25 step size, enabling participants to report papers with small differences in exportisc.
Third, we asked participants to come up with papers that they think may be tricky for existing similarity- computation algorithms. For this, we relied on the commonsense understanding and did not instruct participants on the inner-workings these algorithms. We only provided an example: a naive computa- tion of similarity may think that paper on "Theory of Information Dissemination in Social Networks" has high similority with Information Theory researcher, but in reality, this researcher may not have expertise in reviewing this paper.
Overall, the time needed for participants to contribute to the dataset was estimated to be 5 10 minules. The full instructions of the survey are available in Appendix A
Data release Following the procedure outlined above, we collected responses from 58 researchers. These responses constitute initial version of the dataset that we release in this work. Each entry in the dataset
5

---

Published in Transactions on Machine Learning Research (04/2025)
corresponds to a participant and comprises evaluations of their expertise in reviewing papers of their choice. For each paper and each participant, we provide representations that are sufficient to start working our dataset:
Participant. Each participant is represented by their Semantic Scholar ID and complete bibliography crawled from Semantic Scholar on May 1, 2022.
Paper. Each paper, inchuding papers from participants' bibliographies, represented by its Semantic Scholar ID, title, abstract, list of authors, publication year, and arXiv identifier. Additionally, papers from participants' responses are supplied with links to freely available PDFs (whenever available).
4 Data exploration
In this section we explore the collected data and present various characteristics of the dataset. The su subsequent sections then detail the results of using this data to benchmark various popular similarityccompitatiion algorithms.
4.1 Participants
We begin with discussion of the pool of the survey participants: Table [ displays its key characteristics. First, we note that all participants work in the broad area of computer science and have rich publication profiles (at least two papers published, with the mean of 54 papers and the median 20 papers). many subareas of computer science, including machine learning and artificial intelligence, having two papers is usually sufficient to join the reviewer pool of flagship conferences. Given that approximately 85% of participants cither have PhD are in the process of getting the degrec, we conclude that most of the researchers who contributed to our dalaset are eligible to review for machine learning and artificial intelligence conferences.
Second, we caveat that most of the participants are male researchers affiliated with US-based organizations, with about 40% of all participants being affiliated with Carnegie Mellon University. Hence, the population of participants not necossarily representative of the general population of the machine learning and computer science communities. We encourage researchers to be aware of this limitation when using our dataset. We note that the data collection process does not conclude with the publication of the present paper and we will be updating the dataset as new responses come. We also encourage readers to contribute 5-10 minutes of their time to fill out the survey https://forms. www.glle/PPPPPP 33p1hh8eivG2542hh and make the datasct more representative.
4.2 Papers
We now describe the set of papers that constitute dataset. Overall, participants evaluated their expertise in reviewing 463 unique papers. Out of these 463 papers, 12 papers appeared in reports of two participants, paper was mentioned by three participants, and the remaining papers were mentioned by one participant each.
Table 2 presents several characteristics of the pool of papers in our dataset. First, we note that all but one of the papers are listed on Semantic Scholar, enabling fimilarity-copuuattiin algorithms developed on the dataset to query additional information about the papers from the Semantic Scholar database. For the paper that does not have Semantic Scholar profile, we construct such profile manually and keep this paper in the dataset. Additionally, most of the papers (99%) have their PDFs freely available online, thereby allowing algorithms to use full texts of papers to compute similarities.
According to this tool, 99% of the papers included to our dataset belong to the broad area of computer science- the target area for our data-collection procedure. The remaining four papers belong to the neighboring fields of mathematics, philosophy, and the computational branch of biology. Finally, approximately 75% of all
Semantic Scholar has a built-in tool to identify broad research areas papers (Wade 2022

---

Published in Transactions on Machine Learning Research (04/2025)
TOTAL NUMBER OF PARTICIPANTS: 58
CHARACTERISTIC
QUANTITY
VALUE
GENDER
% MALE
78
COUNTRY
% USA
PHD STUDENT
POSITION
% FACULTY
28
% POST-PRD (NON-PACULTY)
MIN # PUBLICATIONS
2
MAX # PUBLICATIONS
192
EXPERIENCE
MEAN # PUBLICATIONS
54
MEDIAN # PUBLICATIONS
Table 1: Demography of participants. For the first four characteristics, quantities represent percentages of the one or more most popular classes in the dataset. Note that classification done mamually based on publicly available information and may not be free of error. For the last characteristic, quantities are computed based on Semantic Scholar profiles.
TOTAL NUMBER OF PAPERS: 463
CHARACTERISTIC
QUANTITY
VALUE
# ON SEMANTIC SCHOLAR
462
OPEN ACCESS
ON ARXIV
411
# PDF AVAILABLE
457
RESEARCH AREAS
# COMPUTER SCIENCE
459
% BEFORE 2020 2020 OR LATER
25
PUBLICATION YEAR
75
Table 2: Characteristics of the 463 papers in the released dataset. Most of the papers are freely available online and belong to the broad area of computer science.
papers our dataset are published or after 2020, ensuring that our dataset contains recent papers that similaritycommputtioon algorithms encounter in practice.
4.3 Evaluations of expertise
Finally, we proceed to the key aspect of our dataset evaluations of expertise in reviewing the papers reported by participants. All but one participant reported expertise in reviewing at least papers with the mean number of papers per participant being 8.2 and the total number of expertise evaluations being 477
Figure ! provides visualization of expertise evaluations made by participants, First, Figure la displays the counts of expertise values. About 56% of the reported papers have an expertise score greater than equal 4, thereby yielding a good spread.
Second, Figure 1b shows the distributions of pairwise differences in expertise evaluations made by the same reviewer. To construct this figure, for each participant we considered all pairs of papers in their report. Next, we pooled the absolute values of the pairwise differences in expertise across participants. We then plotted the histogram of those differonces in the figure. Observe that the distribution in Figure Tb has a heavy tail, suggesting that our dataset is suitable for evaluating the accuracy similarity-computation methods
TT clarify, the dataset docs not contain the evaluations all participants for each of the 463 unique papers; the data only consistes of the evaluations provided each participant the of papers reported by that participant, making total of 477 evaluations.

---

Published in Transactions on Machine Learning Research (04/2025)
300
250
Count
Count
50
150
100
26
50
2
Difference expertise
Reported experiss
(b) Counts of differences in expertise evalua- tions.
(a) Counts of expertise values reported by participants.
Figure 1: Distribution of expertise scores reported by participants.
both al a coarse level (large differences between the values of expertise) and at fine level (small differences between the values of expertise).

---

Published in Transactions on Machine Learning Research (04/2025)
5 Experimental setup
We now describe the setup of experiments on our dataset.
5.1 Metric
We begin with defining a main metric that we use in this work to evaluate performance of the algorithms. For this, we rely on the Kendall's Tau distance that is closely related to the widely used Kendall's Tau (Kendall 1938) rank correlation coefficient. We introduce the metric and the algorithms this section, followed by the results in Section Subsequently in Section 7 we provide a stratified evaluation that separates out hard and casy instances. Finally in Appendix C evaluate number of additional metries.
Intuition Before we introduce the metric in full details, let us provide some intuition behind it. Consider a pair of papers and a participant who evaluated their expertise in reviewing these papers. We say that a
algorithm makes an error if it fails to correctly predict the paper for which the participant reported the higher value of expertise.
Of course, some pairs of papers are harder to resolve than others (e.g., is harder to resolve papers with expertise scores 4.0 and 4.25 than 4.0 and 1.0). To capture this intuition, whenever algorithm makes an error, we penalize it by the absolute difference in the ezpertise values reported by the participant. Overall, the loss of the algorithm is the sum of losses across all pairs of papers evaluated by participants normalized to take values between and (the lower the better).
Formal definition The intuition behind the metric that we have just introduced should be sufficient to interpret the results of our study the reader may skip the rest of this section and move directly to Section 5.2 However, for the sake of rigor, we now introduce our metric more formally.
Consider any algorithm that produces real-valued predictions of reviewers' expertise in reviewing a given set of papers. We call these predictions the "similarity scores" given by the algorithm. We assume that a higher value similarity score means that the algorithm predicts a better expertise. The metric we define below is agnostic the range or exact values of predicted similarity scores; it only relies on the relative values across different papers.
Now consider any participant r in our study, We let my denote the number of papers for which participant reports their expertise. We let P, {pl (1) (2) pr pr (mm)) denote this of m, papers. For every E {l...., m,), we let (6) {1,1 1.5..... 5) denote the expertise self-reported by participant for paper (i) Next, for every i (l...., m, we let (1) denote the real-valued similarity score given by the algorithm pr
to the pair (reviewer r, paper pr
Having set up this notation, we now define the 'unnormalized' loss of the algorithm with respect to participant r as:
I T((000 + - et)) < x0x
1 - c01 +T(( a x (()) =
20/ of
I,
1
enror
tim
In words, for each pair of papers (pt (()) reported by participant r, the algorithm is not penalized when the ordering of papers induced by the similarity scores {af") (1) , agrees with the ground truth expertise-based ordering (e()) e.))). When two orderings disagree (that is, the algorithm makes an error), the algorithm is penalized by the difference of expertise reported by the participant (|=fa) - =f1) Finally, when scores computed by the algorithm indicate tic while expertise scores are different, the algorithm is penalized by half the difference in expertise (fleff e

---

Published in Transactions on Machine Learning Research (04/2025)
Having the unnormalized loss with respect to a participant defined, now compute the overall loss e [0, of the algorithm. For this, take the sum of unnormalized losses across all participants and normalize this sum by the loss achieved by the adversarial algorithm that reverses the ground-truth ordering of expertise (that is, sets s = -E) and achieves the worst possible performance on the task. More formally,
L
MLr Σ
Σ
Overall, our loss L takes values from O to 1 with lower values indicating better performance.
5.2 Algorithms
In this work, we evaluate several algorithms that we now discuss. All of these algorithms operate with (i) the list of submissions for which similarity scores need to be computed and (ii) reviewers' profiles comprising past publications of reviewers. Conceptually, all algorithms predict reviewers' expertise by ovaluating textual overlap between each submission and papers in each reviewer's publication profile. Let now provide more detail how this idea is implemented in each of the algorithms under consideration.
Trivial baseline First, we consider trivial baseline that ignores the content of submissions and reviewers' profiles when computing the assignment. scores: for each (participant, paper) pair, the TRIVIal algorithm predicts the score to be 1.
Toronto Paper Matching System (TPMS) The TPMS algorithm (Charlin & Zemel 2013 which is based on TF-IDF similarities, is widely used by flagship conferences such as ICML and AAAL. While exact implementation is not publicly available, in our experiments we use an open-source version by Xu et al. (2019) which implements the basic TF-IDF logic of TPMS. We also note that the TF-IDF method was also independently proposed earlier by Price et al. (2010), and subsequently also rediscovered by Kerzendorf et al. (2020).
As technical remark, in our implementation we use reviewers' profiles and all reported papers compute the IDF part of the TF-IDF model. In principle, may be able to get better performance by using larger set of papers from the respective field (e.g., all submissions to the last edition of the conference) to compute IDF.
OpenReview algorithms OpenReview a conference- menference-mmanaemeent system used by machine learning conferences such as NeurIPS and ICLR. It offers family of deep-learning algorithms for measuring expertise of reviewers in reviewing submissions. Tn this work, evaluate the following algorithms which are used to compute affinity scores between submissions and reviewers:
ELMo. This algorithm relies on general- purpose Embeddings from Language Models (Peters et al, 2018) compute textual similaritics between submissions and reviewers' past papers.
general-purpoe
SPECTER. This algorithm employs more specialized document-level embeddings of scientific docu- ments (Cohan et al.) 2020). Specifically, SPECTER explores the citation graph to construct embed- dings that are useful for a variety downstream tasks, including the similarity computation we focus on in this work.
SPECTER+MFF MFRRTER+MFR enhances SPECTER Chang & McCallum 2021 Instead of
constructing a single embedding of cach paper, constructs multiple emboddings that correspond to different facets the paper. These embeddings are then used to compute the similarity scores.
SPECTER2. Finally, SPECTER2 (Singh et al. 2022) an updatod version that builds on top of SPECTER (Chang McCallum 2021). They employ the use of adapters for the embeddings to be more specialized for specific downstream scientific tasks.
10

---

Published in Transactions on Machine Learning Research (04/2025)
We use implementations of these methods that are available on the OpenReview GitHub page and execute them with default parameters.
When implementing these algorithms, an important design decision is how to aggregate or "pool" the similarity scores between a reviewer's profile and a submitted paper. In the openreview toolkit, this pooling is done by first computing the similarity score between the submitted paper and each individual paper the reviewer's profile. These scores are then aggregated into a single similarity measure using pooling method. Common pooling methods include taking the maximum the mean of the individual scores. An alternative, suggested by [Jhih-Y Yi et al. (2024). involves using percentile-sutn as the 75th percentile to improve robustness (as compared to max) against fraudulent behavior. We evaluate all three pooling methods in the context of SPECTER2.
ACL paper matching The Association for Computational Linguistics (ACL) is community of re- searchers working on computational problems involving natural language. The association runs multiple publication vemues, including the flagship ACL conference, and has its own method to compute expertise between papers and reviewers 3 This algorithm trained on 45,309 abstracts of papers found in the ACL anthology (lttps://aclanttholog] org) downloaded in November 2019. The key idea of the model training positive
is split each abstract into two non-contiguous parts and treat two parts of the same abstract example and highly similar parts of different abstracts negative examples. The algorithm uses ideas from the work of WWiting et al. 2019 2022) to learn a simple and efficient similarity function that separates positive examples from negative examples. This function is eventually used to compute similarity scores between papers and reviewers. More details on the ACL algorithm are provided Appendix B
We note that the ACL algorithm is trained on the domain of computational linguistics and hence may suffer from a distribution shift when applied to the general machine learning domain. That said, in line with ELMo, SPECTER, and MFR,, we use the ACL algorithm without additional fine-tuning.
Off-the-Shelf LLMs Large Language Models (LLMs) have increasingly shown high performance on spe- cific downstream tasks. To this end, we evaluate commonly used LLMs-O1-MINI CLAUDE SONNET 3.5, and GEMINI 2 FLASH-OP on this dataset score to any given reviewer-papee paper pair. The specific details are outlined in Appendix
X We query the models with prompi that asks the model to assign
6 Results
In this section, we report the results of evaluation of algorithms described in Section 5.2 First, we juxtapose the TPMS algorithm to explore various aspects of the compluaity-ommpptatioo problem (Section 6.2
all algorithms on our data (Section 6.1 b. Second,
Before presenting the results, we note that in this section, we conduct all evaluations focusing on papers that have PDFs freely available. To this end, we remove 6 papers from the dataset. as they are not freely available online (see Table 2). Similarly, we limit reviewer profiles to papers whose semantic scholar profiles contain links arXiv. One of the participants did not have any such papers, so we exclude them from the dataset.
6.1 Comparison of the algorithms
Our first set of results compares the performance of the existing these algorithms on our data, we need to make some modeling choices faced by conference organizors in practice:
algorithms. To run
Paper representation. First, in their inner-workings, similarity-computttion algorithms operate with some representation of the paper content. Possible choices of representations include: (i) title of the
httpp:://gtthb..com//// http://pwww........oooooowoawwwa
https:///ithhb.. wttp////lloo///eeeeeeeeeeoeeee pawpeeeeeeeepppe-rattteeciieeeee The servers for DeepSeck were consistently overloaded, and Grok3 APIs were unavailable.
11

---

Published in Transactions on Machine Learning Research (04/2025)
paper, (it) title and abstract, and (iii) full text of the paper. We choose option (ii) this option is often used in real conferences and is supported by all algorithms we consider in this work. Thus, to predict expertise, algorithms are provided with the title and abstract of each paper (both papers submitted to the conference and papers in reviewers' publication profiles).
Reviewer profiles. The second important parameter is the choice of papers to include in reviewers proflle. In real conferences, this choice is often left to reviewers who can mamually select the papers they find representative of their expertisc. In our experiments, we construct reviewer profiles automatically by using the 20 most recent papers from their Semantic Scholar profiles. If a reviewer has less than 20 papers published, we include all of them in their profile. Our choice of the reviewer profile size is governed by the observation that the mean length of the reviewer profile in the NeurIPS 2022 conference is 16.5 papers. By setting the maximum number of papers to 20, achieve the mean profile length of 14.8, thereby operating with the amount of information close to that available to algorithms in real conferences.
Statistical aspects To build reviewer profiles, we use publication years to order papers by recency, where we break ties uniformly at random. Thus, the content of reviewer profiles depends on randomness. To average this randomness out, we repeat the procedure of profile construction and similarity prediction 10 times, and report the mean loss over these iterations. That said, we note that the observed variability due to the randomness in the construction of reviewer profiles is negligible, with a standard deviation over all iterations is less than 0.005.
The pointwise performance estimates obtained by the procedure above depend on the selection of participants who contributed to our dataset. To quantify the associated level of uncertainty, we compute 95% confidence intervals as follows. For 1,000 iterations, we create a new reviewer pool by sampling participants with replacement and recomputing the loss of each algorithm on the bootstrapped set of reviewers. To save computation time, we do not reconstruct reviewer profiles for each of these iterations as the uncortainty associated with the construction of reviewer profiles is small. Instead, we reuse profiles constructed to obtain pointwise estimates.
Finally, we also build confidence intervals for the difference in the performance of the algorithms. We do so in addition to the aforementioned procedure since even when the losses of the algorithms fluctuate with the choice of the bootstrapped dataset, the relative difference in performance of a pair of algorithms may be stable. Specifically, we use the procedure above to build confidence intervals for the difference in performance between the TPMS algorithm and each of the more advanced algorithms. TPMS chosen as a bascline for this comparison due to its simplicity.
Results of the comparison Table displays results of the comparison. The first pair of colurns presents the loss of each algorithm on our dataset and the associated confidence intervals. The third and the forth columns investigate the relative difference in performance between the non-trivial algorithms: they display the differences in performance between TPMS and each of the more advanced algorithms (OpenReview algo- rithms and ACL) together with the associated confidence intervals. We make several important observations from Table g
First, we note that all algorithms we consider this work considerably outperform the TRIVIAL baseline, confirming that content of papers is useful in evaluating expertise.
Second, comparing the four algorithms from the OpenReview toolkit, we note that the SPECTER family of algorithms outperform E.LMo. The former rely on domain-specific embeddingg[) while ELMO uses general-purpose embeddings. Thus, the nature of the text similarity computation task in the academic context may be sufficiently different from that in other domains.
Third, we note that under modeling choices (paper representation and length of reviewers' profiles) that mimic choices that have been made in real conferences, the SPECTER2 algorithms performs the best, with SPECTER2 with max pooling achieving the lowest loss.
8 MPP SPECTER, and SPECTER2 are initialized with ScBERT (eeltgy al. 2019
12

---

Published in Transactions on Machine Learning Research (04/2025)
ALGORITHM
Loss
95% CT FOR Loss
WITH TPMS
95% CT FOR A
TRIVIAL
0.50
TPMS
0.28
[0.23, 0.33]
ELMO
0.35
[0.31, 0.42]
+0.07
[0.02, 0.1
SPECTER
0.27
[0.21, 0.34
-0.01
0.006 0.04
MFR
0.24
[0.18, 0.30
-0.04
-0.09,
SPECTER2 MEAN PoOL
0.25
[0.19, 0.30
-0.03
0.02
SPECTER2 75TH PoOL
0.24
[0.19, 0.30
0.04
[-0.1,
SPECTER2 MAX POOL
0.22
[0.18, 0.26]
-0.06
[-0.11, 0.022
ACL
0.30
[0.25, 0.36]
+0.02
1-0.02.0007
01-MINI
0.25
(0.21, 0.29]
-0.03
[-0.07, 0.0]
CLAUDE SONNET 3.5
0.31
[0.27, 0.35
+0.03
[-0.11, 0.07]
GEMINI 2 FLASH
0.38
[0.34, 0.41]
+0.10
[0.06, 0.14]
algorithms on the collected data. All algorithms operate with reviewer profiles consisting of the 20 most recent papers and use titles and abstracts of papers. Lower values of loss are better. A positive (respoctively, negative) value of A indicates that the algorithm performs worse (respectively, better) than TPMS.*
Table 3: Comparison of
*Tater in Table evaluate TPMS algorithm when full texts of papers additionally provided and observe that TPMS closes the gap with SPECTER2 when using this information.
Fourth, we note that generic LLMs have large variation of performance. While o1-MINI performs on par with the SPECTER2 algorithm, GEMINI 2 FLASH the worst performing algorithm barring the TRivIal baseline.
Fifth, among the pooling options in SPECTER2, max pooling delivers the best performance, while the 75th percentile pooling method performs nearly as well.
The sixth finding is the most surprising. Observe that the TPMS algorithm much simpler than other non-trivial algorithms: in contrast to ELMo, SPECTER, ACL, and the LLMs, it does not rely on carefully learned embeddings. However, TPMS is competitive against complex SPECTER2 and SFECTER+MFR and even outperforms CLAUDE SONNET 3.5 and ACL. Moreover, TPMS is the only algorithm in the OpenReview toolkit that can make use the full text the papers. In Section 6.2 we find that when TPMS uses the full text of all papers (including reviewers" past papers), it closes the gap with SPECTER2.
Finally, we note that some of the confidence intervals for the performance of the different algorithms, well for the relative differences, are overlapping. It is, therefore, crucial to increase the size of our dataset to enable more fine-grained comparisons between the algorithms.
6.2
The role of modeling choices
In the beginning of Section 6.1 we made two modeling choices pertaining to (i) representations of the papers provided ciillarity-oomputatii algorithms, and (ii) the size of reviewers' profiles used by these algorithms. In this section, investigate these two questions in more detail.
algorithms designed to work with titles and/or abstracts of papers (c.g., SPECTER) while others can also incorporate the full texts of the manuscripts (e.g., TPMS). Consequently, there is a palential trade-off between accuracy and compu- tation time. Richer representations are envisaged to result in higher accuracy, but are also associated with an increasod demand for computational power. As with the choice of the algorithm itself, there is no guidance on what amount of information should be supplied to the algorithm as the gains from using more information not quantified. With this motivation, first question is:
Question 1 (paper representation). Some
13

---

Published in Transactions on Machine Learning Research (04/2025)
0.4
Loss
Paper content
title
title +abstract
full text
0.0
10
Number of papers in reviewer profile
Figure 2: Impact of different choices of parameters the quality of predicted similarities. Confidence intervals are not shown (sce Table 4
What are the benefits of providing richer representations of papers to similarity- computation algorithms?
Question 2 (reviewer profile). The second important. choice the size of reviewers' profiles. On the one hand, by including only very recent papers in reviewers' profiles, conference organizers are at risk of not using enough data, to obtaín accurate values of expertise. On the other hand, old papers may not accurately represent the current expertise of researchers and hence may result in noise when used to compute expertise. Thus, our second question is:
What is the optimal number of the most recent papers to include in the profiles of reviewers? ?
To investigate these questions, we choose the TPMS algorithm as the workhorse to perform experiments. We make this choice for reasons. First, TPMS can work with all possible choices of the paper representation: title only, title and abstract, and full text of the paper. In contrast, other methods do not support using the full text of the papers. Second, TPMS is fast to execute, enabling us to compute expertise for hundreds of configurations in a reasonable time.
With the algorithm chosen, we vary the mumber papers in the reviewers? profiles from to 20. For each value of the profile length, we consider three representations of the paper content: title, tabscctbbstract, and full text of the paper. For each combination of parameters, similarities using the approach introduced in Section 6.1 The only exception is that repeal. the procedure for averaging out the randomness in the profile creation times (instead of 10) to save the computation time.
construct reviewer profiles and predict
Results Figure and Table ! provide answers to the questions we study in this section. Figure 2 shows the pointwise loss the TPMS algorithm for each choice of parameters. To save computation time, we do not build confidence intervals for each combination of parameters. Instead, Table 4 sets the number of papers in reviewers' profiles to 20 (consistent with Table 3 and presents confidence intervals for losses incurred by the algorithun under different choices of paper representations. We now make two observations.
First, paper abstracts very useful in improving the quality of expertise prediction compared to titles alone (the improvement of 0.07). Including full texts of the papers in reviewers' and papers' profiles results in additional improvement (0.03). Overall, ignoring the minor differences in the datasets between this section and Section [..1, we observe that TPMS with titles, abstracts, and full texts of papers demonstrates a performance quite close SPECTER2 which cannot handle the full texts of papers. Thus, it may be of interest to develop modern deep-learning based algorithms with longer contexts to incorporate full texts of papers, and further boost performance on the similarity-compuatiian task.
14

---

Published in Transactions on Machine Learning Research (04/2025)
PAPER REPRESENTATION
Loss
95% CT FOR Loss
WITH ARSTRRTT
95% CT POR A
TITLE
0.33
[0.29, 0.38]
+0.07
00.02, 0.12]
TITLE- ABSTRACT
0.27
[0.22, 0.32
FULL TEXT
0.24
[0.19, 0.30]
-0.03
[- 0-009,
Table 4: Performance of the TPMS algorithm with 20 most recent papers included in reviewers' profiles and with different choices of the paper representation. Lower values of loss are better. A positive (respectively, negative) value of A indicates that the corresponding choice of the paper representation lcads to weaker (respectively, stronger) performance.
Second, the loss curves plateau once reviewer profiles include 8 or more of their most recent papers. Additional increase of the profile length does not impact the quality of predictions. Thus, in practice, reviewers may be instructed to include 10 representative papers to their profile, which for many active researchers amounts to the number papers published in 1-3 years.
7 Easy versus hard stratification
Recall that our loss metric (see Section assigns different weights errors of an algorithm: errors on pairs with similar values of reviewer's expertise are penalized less than errors on well-separated pairs. The overall performance of the algorithm is then captured in single number (loss) which does not characterize the type of mistakes made by the algorithm.
To provide deeper characterization of the algorithm performance, we now conduct additional evaluations. In these evaluations, we focus on two important regions of the similarity-computatie problem. Specifically, from all expertise evaluations made by participants, we select two groups of triplets, where each triplet consists of a participant and two papers (P1.P2) evaluated by this participant:
"Easy" triplets. The first group comprises 261 triplets where participant reported high expertise (greater than equal to four) in one paper and low expertise (less than or equal to two) in another.
We call these triplets "easy" the gap in the expertise between the two papers is large. Note that in real conferences it is important to resolve the casy triplcts correctly to ensure that reviewers do not get assigned irrelevant papers.
"Hard" triplets. The second group comprises 417 triplets where a participant reported high expertise (greater than or equal to four) in both papers and the values of expertise are different for the two papers.
We call these triplets "hard" as the gap in the expertise between the papers is small. That said, we note that resolving hard triplets in practice is also very important: in real conferences, we want to assign papers to the most suitable reviewers and we need to be able to distinguish two papers with high but not equal values of expertise to achieve this goal.
Note that by focusing on these two groups, exclude regions of the problem which may be considered less important. For example, the ability of an algorithm to correctly order two low-expertise papers is less crucial as long as the algorithm can reliably distinguish these papers from the high-expertise papers.
We now study the performance the algorithms introduced in Section 5.2 on the easy and hard groups of triplets. For all algorithms we use titles and abstracts of papers as the data source and include 20 papers in the reviewers' profiles. Specifically, for each triplet we compare the ordering of papers predicted by an algorithm with the expertise-induced ordering. We then compute the accuracy of each algorithm the fraction of triplets correctly resolved by the algorithm (from 0 to 1, larger values are better). In addition, we also evaluate TPMS in the full text regime to estimate the effect of papers' texts on the accuracy of the similarity-ompuutttinn algorithms
6TPMS in the full text regime is evaluated slightly different dataset as explained Section 6.2
15

---

Published in Transactions on Machine Learning Research (04/2025)
EASY TRIPLETS (n = 261)
HARD TRIPLETS (n = 417)
ALGORITHM
ACCURACY
95% C1
ACCURACY
95% C1
TPMS (T+A)
0.80
(0.72, 0.87]
0.62
(0.54, 0.69)
TPMS (FULL TEXT)
0.84
0.76,, 0.91
0.64
(0.56, 0.70
ELMo
0.70
[0.62, 0.78
0.57
[0.51, 0.63
SPECTER
0.85
[0.76, 0.92
0.57
[0.50, 0.63
SPECTER+MFR
0.88
00.81, 0.94
0.60
00533; 0.66
SPECTER2 MEAN PoOL
0.84
0.73,, 0.92
0.59
0.53, 0.65
SPECTER2 75TH POOL
0.85
[0.75, 0.93
0.60
[0.54, 0.65
SPECTER2 MAX POOL
0.89
[0.82, 0.94
0.61
[0.54, 0.67
ACL
0.78
[0.69; 0.86]
0.62
[0.55; 0.68)
01-MINI
0.82
0.755, 0.88
0.52
[0.47, 0.57
CLAUDE SONNET 3.5
0.72
[0.66, 0.77
0.52
(0.47, 0.58
GEMINI 2 FLASH
0.57
[0.50, 0.63
0.41
[0.38, 0.43
Table 5: Results of additional evaluations. Higher values of accuracy are better. SPECTER2 demonstrates the best performance on easy triplets and TPMS (Full Text) has the highest accuracy on hard triplets.
Table 5 demonstrates the results of additional evaluations. First, note that all algorithms have moderate to high accuracy in resolving the easy triplets. Most algorithms are able to detect papers that reviewers have low expertise with probability close to above 80,, with SPECTER2 reaching nearly 90% accuracy. Surprisingly, LLMs GEMINI 2 FLASH and CLAUDE SONNET 3.5, along with ELMo are the 3 with the lowest performance, with GEMINI FLASH having low 57% accuracy. However, we note that in non-trivial amount of cases (more than 10%), the algorithms are unable to distinguish paper that reviewer is well- qualified to review versus a paper the reviewer does not have necessary background to evaluate. Thus, there is a vital need to improve the similarity-compputtion algorithms.
Getting to the hard triplets, we observe a significant drop in performance with most algorithms being a bit better than random: the best-performing algorithm (TPMS) reaches 62% accuracy in the title-abstracc regime and 64% in the full text regime. Perhaps surprisingly, while the SPECTER family of algorithms outperform TPMS easy triplets, they fare slightly worse than TPMS hard triplets (we caveat though that error bars are too wide to make decisive comparison). Furthermore, the performance of all three LLMs are no better than random, with GEMINI 2 FLASH performing worse than random. This observation suggests that there may be a value in additionally fine-tuning these advanced algorithms on hard triplets to improve their practical performance.
Finally, we observe that on both easy and hard triplets, full texts of the papers are instrumental in improving performance of the TPMS algorithm. This observation supports our intuition that mmilaaiiy-computation algorithms do indeed benefit from employing full texts of papers.
With these remarks, conclude evaluations of algorithms the dataset we collected.
8 Discussion
In this work, we collect novel dataset of reviewers' expertise in reviewing papers. In contrast to datasets collected in previous works, our dataset (i) is released publicly, and (ii) contains evaluations of expertise made by scientists who have actually read the papers for their own research purposes. We use this dataset to compare several existing expertise-estimaaiin algorithms and help venue organizers in choosing an algorithm in an evidence-based manner.
Most importantly, we find that current algorithms make large number of errors. Given the growing number of submissions in many fields of research, the need for automation in reviewer assignments is only growing.
16

---

Published in Transactions on Machine Learning Research (04/2025)
There is thus urgent and vital need to develop significantly improved algorithms to match reviewers to papers, thereby in turn making the peer-review process considerably better.
Our dataset can be used to develop as well as evaluate new expertise-estimation algorithms. We encourage researchers from the natural language processing and other communtiies to use our data in order to improve peer review.
Limitations. Finally, we mention several caveats that researchers should be aware of when working with our dataset and interpreting the results of our experiments. First, our dataset comprises evaluations of expertisc in reviewing papers that were written some time ago. In contrast, in real conferences, many papers are recent and not available online. Thus, incoming citations to papers included in our dataset may constitute information that is not available to the algorithms in real life. While the algorith we evaluate in this work do not rely on the citation relationship, this caveat may important for future work.
Second, the experiments we conduct in this work rely on Semantic Scholar profiles. These profiles are constructed automatically and may not be accurate. Thus, mistakes of the algorithms observe in this work can be partially due to the noise in the profile creation.
Third, the design of the survey interface can inffuence the data and, consequently, the evaluations. We hope for further studies on reviewer expertise, potentially employing diverse methodologies. When combined with our dataset, these additional studies could provide a more robust and multi-dimennsonal evaluation.
Finally, we reiterate that the present version of the dataset can have sampling bias and may be constructed by participants who collectively not representative of the general computer science community. For example, about 40% of participants are affiliated with Carnegie Mellon University. To alleviate this issue, we continue the data collection process and encourage the readers of this paper to contribute their dala points to the datasst:
https://forms
Acknowledgments
We sincerely thank all participants of our survey for their contribution to the datasct. This work was supported by NSF CAREER 1942124 and NSF 1763734.
References
Omer Anjum, Hongyu Gong, Suma Bhat, Wen-Mei Hwu, and JinJun Xiong. PaRe: A paper-reviewer match- ing approach using common topic space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP) pp. 518 528, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ /t///////11////// URL
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Melhods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL)), pp. 3615-3620, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: (1/86633v1/DD/ 1371. URL httpp:://eclaataaooooo.o///////p
Federico Bianchi and Flaminio Squazzoni. Is three better than onc? Simulating the effect of reviewer selection and behavior on the quality and elficiency of peer review. In Proceedings of the 2015 Winter Simulation Conference, pp. 4081-4089. IEFF Press, 2015.
Nick Black, Susan Van Rooyen, Fiona Godlee, Richard Smith, and Stephen Evans. What makes a good reviewer and good review for general medical journal? Jama, 280(3):231122)) 1998.
Niclas Bochmer, Robert Bredereck, and André Nichterlein. Combating collusion rings hard but possible. arXiv preprint ar Xio:2112. 08444, 2021.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Haw-Shiuan Chang and Andrew McCallum. Cold-start paper recommendation using multi-facet embed- ding. Overleaf preprint, 2021.
http:///cccc////ffaaaa/22220000. [Last
Accessed: 3/15/2023].
L. Charlin and R. S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system, 2013. URL http::/wwer http:://ww.c.... toronto..... wwwp://eeelllbbccuettttta//a///e
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. Specter: Document-level representation learning using citation-informed transformers, 2020.
Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via partitioning: The price in terms evaluators' expertise. In HCOMP, 2022.
Susan Dumais and Jakob Nielsen. Automating the assignment of submitted manuscripts to reviewers. Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information relrieval, pp. 233 244, 1992.
T Fiez, N Shah, and Ratliff. A SUPER* algorithm to optimize paper bidding in peer review, In Conference on Uncertainty in Artificial Intelligence, 2020.
Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre. Assigning papers to referees. Algorithemica, 58(1):119- 136, Sep 2010. ISSN 1432-0541. doi: 10.1007/s00455--099 000-0986----00 URL https://doi.or http///oo.orr//////// 100 L007/s00453-009-93888
Judy Goldsmith and Robert Н. Sloan. The Al conference paper assignment problem. AAAI Workshop Technical Report, WS-07-10:53 57, 12 2007.
Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NewrIPS, 2020.
Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, and Fei Fang. A dataset on malicious paper bidding in peer review, 2022. URL https://aaxii: org/ 2ttp://227..2220
Jh,h-Yi, Hsieh, Aditi Raghunathan, and Nihar B. Shah. Vulnerability of texi-matching in ml/ai conference reviewer assignments to collusions. (arXiv:2412.06606) December 2024. doi: 10.48550/arXiv 21.2022222222.00066.. 06606 arXiv:2412.06606.
URL http://arxiv. www.w.rgggsb.........
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):88 199, 1938.
Wolfgang E. Kerzendorf, Ferdinando Patat, Dominic Bordelon, Glenn van de Ven, and Tyler A. Pritchard. Distributed peer review enhanced with natural language processing and machine lcarning, 2020.
Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local lairness constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 36-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIs 20122222 URL https: htttps:///cllii //apaan///ca. org/D18-201 2012
Kevin Meylon-Brownr Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, Dinesh Raghu, et al. Matching papers and reviewers at large conferences. arXiv preprint Xix:22202. 12279, 2022.
Lei Li, Yan Wang, Guanfeng Liu, Meng Wang, and Xindong Wu. Context-aware reviewer assignment for trust enhanced peer review. PLOS ONE, 10(6):1-28, 06 2015. doi: 10. 10. fo..7///jornall.poe.011000 URL
0130493
18

---

Published in Transactions on Machine Learning Research (04/2025)
Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recommender Systems, RecSys '14, pp. 25 25-32, New York, NY, USA, 2014. ACM. ISBN 778-1-4503-2668-1 doi: 10.1145/22445112244577999 URL http://doi. acm org/10. 1145/ 2645710 2645749
Cheng Long, Raymond Wong, Yu Peng, and Liangliang Ye. On good and fair paper-reviewer assignment. In Proceedings IEEE International Conference on Dala Mining, ICDM, pp- 1145-1150, 12 2013. ISBN
578-0-7695--105--.
Reshef Meir, Jérôme Lang, Julien Lesca, Natan Kaminsky, and Nicholas Mattei. A market-inspired bidding schome for peer review paper assignment. In Games, Agents, and Incentives Workshop at AAMAS, 2020.
Robert K Merton. The Matthow effect in science. Science, 159:56-63, 1968.
David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, KDD '07, pp. 500 509, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-609-7. doi:
URL http://doi...... acm. org/10. 11..0.114//12119900 1281247
OpenReview. Paper-reviewer affinity modcling for openreviow. https:://github.com/ cww com/openreveeee/
cewereviee--xperttae 2022.
Justin Payan. Fair allocation problems in reviewer assignment. In Proceedings of the 21sl International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 1857-1859, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392138
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pp. 2227 2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. (4.18653/v/ /11///////1/// URL https://acllanthologyy http://aaaaanooooooooooo////----
Simon Price, Peter A Flach, and Sebastian Spiegler. Subsift: a novel application of the vector space model to support the academic research process. In Proceedings of the First Workshop on Applications of Pattern Analysis, pp. 20-27. PMLR, 2010.
Marko A. Rodriguez and Johan Bollen. An algorithm to determine poer-reviewers. In Proceedings of the 1 17th ACM Conference Information and Knowledge Management, CIKM '08, pp. 319 328, New York, NY, USA, 2008. ACM. ISBN 978-1-59593-991-3 doi: 00.1145/1458052.145588 URL http://doi..... acm. org/ 10. 1145/ 114455/148082.155555 1458127
Nihar B. Shah. Randomized transparency to mitigate collusions, August 2022a. URL https://
researchonresearch. htt.a///caeaaae////////////aaaaa
Nihar B Shah. An overview of challenges, experiments, and compulational solutions in peer review.
Commuicatiins of the ACM. Extended version at https://www.c.. cmu. wwt..cm..m.emiiiieeee htt..eedda/mhharrrrrrptttt SurveyPeerKeviee.pde 2022b.
Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulriee Von Luxburg. Design and analysis of the NIPS 2016 review process. The Journal of Machine Learning Research, 19(1):1913 1946, 2018.
Amanpreet Singh, Mike D' D'Aryy, Arman Cohan, Doug Downey, and Sergey Feldman. Scirepeval: A multi- format benchmark for scientific document representations. In Conference on Empirical Methods in Natural Language Processing, 2022. URL hhtpp:://app www.sti /tco htt.:/cccoaaaa.oeerrrorrrp/ii/ss 224018117
Flaminio Squazzoni and Claudio Gandelli. Saint Matthew strikes again: An agent- based model of peer review and the scientific community structure. Journal of Informetrics, 6(2):265-25) 275, 2012.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Ivan Stelmakh, Nihar Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. Journal of Machine Learning Research, 22(163):1-00) -6, 2021. URL http:://jnrr.rrgg//eeepe
v22/20-190.nc
Mao Tan, Zhuocen Dai, Yuling Ren, Toby Walsh, and Martin Aleksandrov. Minimal-envy conference paper assignment: Formulation and a fast iterative algorithm. In 2021 5th Asian Conference on Artificial Intelligence Technology (ACAIT), pp. 367-674, 2021. doi: (000///AAAT33529920000
Wenbin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint- optimization. In Proceedings of the 2010 GEE//WCC/AMM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 01, WI-IAT '10, pp. 34 41, Washington, DC, USA, 2010. IEEE Computer URL http://dxx doi http://d...oi..r/1.....o..
Society. ISBN 778-0-7695-4191-4
doi:
WI- IL-TAT.WA 2010. 133
Warren Thorngate and Wahida Chowdhury. By the numbers: Track record, flawed reviews, journal space, and the fate of talented authors. In Advances in Social Sinulation, pp. 177-188. Springer, 2014.
Stefan Thurner and Rudolf Hanel. Peer-review in world with rational scientists: Toward selection of the average. The European Physical Journal B, 84(4):707-71 711, 2011.
D. Tran, G. Cabanac, and G. Hubert. Expert suggestion for conference program committees. Tn 2017 11th International Conference on Reseorch Challenges in Information Science (RCIS), pp. 221-232, May 2017. doi: 1LL109/RCIS.001.779550
Chris Triggle and David Triggle. What is the future of peer review? why is there fraud in science? is plagiarism out of control? why do scientists do bad things? is it all a case of: all that is necessary for the triumph of evil is that good men do nothing?". Vascular health and risk management, 3:39-53, 02 2007.
Alex D. Wade. The semantic scholar academic graph (S2AG). In Companion Proceedings of the Web Conference 2022 (WWW'22 Companion), 2022.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg- Kirkparikp. Simple and effective paraphras- tic similarity from parallel translations. In Proceedings of the 57th Annual Meeling of the Association for Computational Linguistics, pp. 4602 4608, Florence, Italy, July 2019. Association for Computational Lin- guistics. doi: (0.18633vv1/PH5 1453. URL
John Wieling, Kevin Gimpel, Graham Neubig, and Taylor Berg-kirkpatrick Paraphrastic representations scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 379 388, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics. URL https://aclanthene wrwpollooyyoog20/00 org/2022 emnlp-demos.33 38
Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens Van Der Maaten, and Kilian Weinberger. Making paper reviewing robust to bid manipulation attacks. In Marina Meila and Tong Zhang (eds.); Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11240-11250. PMLR, 18 18-24 Jul 2021. URL URL http::://proceeednggs mlr.
preas/r139/mu21b.ht
Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. On strategyproof conference review. In IJCAT, 2019.
Yue Zhao, Ajay Anand, and Gaurav Sharma. Reviewer recommendations using document vector embeddings and a publisher database: Implementation and evaluation. IEEE Access, 10:21798 798-2181,, 2022.
Appendices
20

---

Published in Transactions on Machine Learning Research (04/2025)
A More details on the survey used for data collection
In this section we provide full instructions that were given to the participants of our data-collection procedure.
Dataset of Reviewing Expertise
The goal of this experiment is to collect dataset to help researchers design better algorithms for computing similarities between papers and reviewers: these algorithms will help to improve matching of reviewers to papers in many conferences like NeurIPS, AAAI, ACL, etc.
WHAT YOU NEED DO?
***Recall 5-10 papers in your broad research area that you read to reasonable extent in the last year and tell your expertise in reviewing these papers. *** (HINT: To quickly recall what papers you read recently, you can search for arxiv.org org or an analogous website in your browser history.)
WHICH PAPERS TO REPORT?
Papers should not be authored by you
Papers should be freely available online (preferably arXiv, but other open sources are also fine)
Suggesstioss**
Try to choose a set of papers such that some pairs are well well-separated and some are very close in terms of your expertise
Please try to avoid ties in the expertise ratings you provide
Try to think of some papers that are less famous to make the dataset more diverse
examples that not obvious and may be tricky for the similarity--oopttttiion algorithms. For example, naive computation of similarity may think that paper on "Theory of Information Dissemination in Social Networks" has high similarity with an Information Theory researcher, but in reality, this researcher may not have expertise in reviewing this paper
Try to provide
WHAT PARTS OF DATA YOU PROVIDE WILL BE RELEASED?
To facilitate the development of better algorithms for similarity computation (trained to perform well on your data!), we will publicly release data collected this survey (except email addresses) in non-anonymized form. Your email will not be released.
WHO IS RUNNING THIS SURVEY?
The survey is organized by Graham Neubig, Mihar Shah, Ivan Stelmakh, Sarina Xi (CMU), and John
Wieting (CMU -> Google Research). Contact Ivan at stivêcs edu if you have any questions.
[...]
List up to 10 papers in your broad research area that you read to reasonable extent in the last year and tell us your expertise in reviewing these papers. Please try to enter at least papers.
21

---

Published in Transactions on Machine Learning Research (04/2025)
Link to Paper 1:
Expertise in reviewing Paper 1:
1.0 (I am not qualified to review this paper)
1.25
1.5
1.75
2.0 (I can review some aspects the paper, but can't make a reliable overall judgment)
3.0 (I can provide an adequate review, but substantial part of the paper is outside of my expertise)
4.0 (I have background in most aspects of the paper, but some minor aspects are beyond my expertise)
5.0 (I have background necessary to evaluate all the aspects of the paper)
[...]
B
More details on the ACL algorithm
In this section, provide more details the ACL algorithm that evaluate in this paper.
Training The model is trained on a large corpus of 45,309 abstracts from the ACI anthology and is inspired by the work of [Wieting et al. (2019 2022). Specifically, the model optimizes a max-margin contrastive learning objective which is defined as follows. First, each abstract a; from the corpus is split into two disjoint segments of text uniformly at random. These segments are then uniformly at random allocated into two equally-sized groups: a:" E A1 and (2) E A2.
Second, positive and negative examples are constructed as follows:
Positive ezample: For each abstract az pair (a. a constitutes a positive example.
Negalôve example: For each passage dj (1) E Aı, counterpart, 4, + af?) from A2 is selected to maximize the notion of cosine similarity
fo fe(a () = cos (g(a(" g)) [
where g is the sentence encoder with parameters 0. Pair (a, (1) t4) constitutes negative example.
Finally, with this procodure to build positive and nogative examples, the objective of the ACL algorithm is defined as:
min Σ max {o- (0-5 aq()) fo(0," ,4),, 0).
Inner-g of the algorithm relies on sentencepiece embeddings Kudo & Richardson, 2018 https: //gthuub.
with dimension of 1,024 and vocabulary size of 20,000. The encoder, 9, simply mcan pools the learned sentencepiece embeddings, making for efficient encoding, even on CPU (see Wieting et al. 2019 2022 for more details on encoding speed). In the training procedure, a batch size of 64 is used and the model is trained for 20 epochs. The margin, 6, is set to 0.4.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Inference At the inference stage, for given pair of a submission and reviewer, the similarity score is defined as combination of cosine similarities between the submission's abstract and three most-similar abstracts from the reviewer's profile. Specifically, let $1, and 83 be the top-3 cosine similarities between the submission's 's abstract and abstracts from the reviewer's profile. The similarity score between the submission and the reviewer then defined as follows:
82
83 3
81
If reviewer has less than three abstracts in the profile, the corresponding cosine similarity scores 8, are set to be zero.
C Additional metrics
For completeness, we evaluate the algorithms several additional metrics. All of these additional evaluations are done on using titles and abstracts of the papers. In addition to the weighted Kendall Tau Distance (WKTD) 5.1.. we use WKTD@k, Mean Reciprocal Rank (MRR)@K, Normalized Discounted Cumulative Gain (NDCG)&k and Precision@k. In order to define these metrics formally, we introduce some notation. The metrics operate separately over each participant in the dataset and over cach algorithm, and hence are defined below for any individual participant and any individual algorithın.
The top k papers as ranked by the participant in the dataset is represented as the ordered set Ra- The set all papers for the participant denoted as R.
The set of paper pairs (p:,P)) where ps E Rg Pj E Rs is denoted as R2.
The top papers ranked by the algorithm is represented as the ordered Ag-
The expertise score for a paper provided by the participant in the dataset is denoted as e(p), And let Aci- = e(Pi) clPs).
The rank of a paper given by the algorithm is denoted p(P).
The similarity score determined by an algorithm for paper p denoted s(x), And let Asis ((P)) - ((,))
With this notation in place, the additional metrics we evaluate are defined as follows:
WKTD@k
Σ
(H{Asú x Aei <0} x Deig + H(Asi x Aeg = 0} x o Aeig))
(PEP))ER
MRR@k Σ (p)
PER, DCG IDCG
(P)
(P) log, + 1)
NDCG@k
where DCG@k
Σ PEAr
IDCG@k
Σ
log, (i + 1))
Pe Rs.
@rk
R& Asl.
For MRR, NDCG, and Precision, we take the mean across all participants for each algorithm as the score. For WKTD, we normalize similar to as described in Section 5.1 where the subscript r represents each participant:
Σ, WKTD@K
WKTD@k D@knorm
Σ. Eipp
We further note that for WKTD, the lower the value the better. For MRR, NDCG, and Precision, the higher the value the better. The results are presented in Tables 6 7
and
23

---

Published in Transactions on Machine Learning Research (04/2025)
ALGORITHM
WEIGHTED KENDALL TAU DISTANCE@1
MRR@1
NDCG@1
PRECISIONG1
TPMS
0.29
0.5
0.86
0.24
ELMO
0.34
0.41
0.84
0.17
SPECTER
0.23
0.46
0.88
0.24
SPECTER+ MFR
0.16
0.48
0.91
0.27
SPECTER2 MEAN PoOL
0.25
0.44
0.87
0.17
SPECTER2 75TH POOL
0.21
0.47
0.88
0.19
SPECTER2 MAX POOL
0.17
0.50
0.90
0.33
ACL
0.29
0.48
0.86
0.24
01-MINI
0.22
0.55
0.89
0.34
CLAUDE SONNET 3.5
0.22
0.53
0.89
0.33
GEMINT FLASH
0.36
0.46
0.84
0.24
Table 6: Comparison of similarity-compputtion algorithms on the collected data using different metrics @top 1. All algorithms operate with reviewer profiles consisting of the 20 most recent papers and titles and abstracts of papers. Best values are bolded for each column.
ALGORITHM
WRIGHTED KENDALL TAU DISTANCE&3
MRR@3
NDCG@3
PRECISION@3
TPMS
0.25
0.41
0.89
0.53
ELMO
0.35
0.39
0.85
0.47
SPECTER
0.25
0.43
0.89
0.55
SPECTER MFF
0.23
0.44
0.90
0.58
SPECTER2 MEAN PoOL
0.22
0.43
0.90
0.57
SPECTER2 75TH PoOl
0.19
0.44
0.91
0.58
SPECTER2 MAX POOL
0.19
0.46
0.91
0.59
ACL
0.26
0.41
0.88
0.54
01-MINI
0.22
0.44
0.91
0.58
CLAUDE SONNET 3.5
0.27
0.43
0.89
0.54
GEMINI FLASH
0.37
0.40
0.86
0.48
Table 7: Comparison of
algorithms on the collected data using different metrics @top 3. All algorithms operate with reviewer profiles consisting of the 20 most recent papers and use titles and abstracts of papers. Best values are bolded for each column.
24

---

Published in Transactions on Machine Learning Research (04/2025)
ALGORITHM
WEIGHTED KENDALL TAU DISTANCE&S
NDCG@5 MRR@5
PRECISIONaS
TPMS
0.27
0.38
0.91
0.72
ELMO
0.34
0,37
0.89
0.70
SPECTER
0.25
0.39
0.92
0.75
SPECTER+ MFR
0.22
0.40
0.93
0.77
SPECTER2 MEAN PoOL
0.23
0.39
0.92
0.76
SPECTER2 75TH POOL
0.22
0.39
0.92
0.75
SPECTER2 MAX POOL
0.20
0.40
0.93
0.77
ACL
0.27
0.39
0.91
0.74
O1-MINI
0.23
0,40
0.93
0.76
CLAUDE SONNET 3.5
0.29
0.39
0.91
0.72
GEMINI 2 FLASH
0.38
0.37
0.89
0.70
Table 8: Comparison of similarity-comppuaaiii algorithms the collected data using different metrics @top 5. All algorithms operate with reviewer profiles consisting of the 20 most recent papers and use titles and abstracts of papers. Best values are bolded for each column.
ALGORITIM
WEIGHTED KENDALL TAU DISTANCER7
MRR@7
NDCG@7
PRECISION@7
TPMS
0.28
0.36
0.93
0.81
ELMO
0.35
0.36
0.91
0.80
SPECTER
0.27
0.37
0.93
0.81
MFR
0.23
0.37
0.94
0.82
SPECTER2 MEAN POOL
0.24
0.37
0.94
0.83
SPECTER2 75TH Poor
0.23
0.37
0.94
0.82
SPECTER2 MAX PoOl
0.21
0.37
0.95
0.82
ACL
0.29
0.36
0.92
0.80
O1-MINI
0.24
0.37
0.94
0.82
CLAUDE SONNET 3.5
0.30
0.37
0.93
0.81
GEMINI FLASH
0.38
0.36
0.91
0.80
algorithms on the collected data using different metrics @top 7. All algorithms operate with reviewer profiles consisting of the 20 most recent papers and use titles and racts of papers. Best. values are bolded for each column.
Table 9: Comparison
D LLM Prompt
Here is the prompt we used to evaluate LLMs.
You are an expert in assigning reviewers to papers a top machine learning conference. You want to make sure that the reviewers can give high quality and relevant reviews the paper they assigned.
You are given the following input:
"Reviever papers", which is a selection of the most recent papers by the reviewer, containing the title and abstract of each paper.
"New paper", which contains the title and abstract of a paper that under consideration for assignment.
Your job is to assign score, which is represented by integer between 0 to 100, that evaluates how well the new paper matches with the reviewer's expertise based all of their "Reviewer papers".
25

---

Published in Transactions on Machine Learning Research (04/2025)
Consider the following criteria:
1. The score should be based on the similarity between the new paper and the reviewer papers.
2. The score should be higher if the reviewer has more background knowledge and expertise in new paper.
The score should lower the reviewer less background knowledge and expertise in new paper.
Please provide the score for the new paper exactly in the following format, where the content in <> is what are generating. Do not provide any additional text and do not include the <>: <score>
Explanation: <briefly explain why you gave <score>
NOTE:
The input is in >
- Do no hallucinate in the output and generate information that is not present in the
reviewer or new papers, make sure to explain why you scored a certain value.
Input:
Reviewer papers:
1. Title: #title*
Abstract: *abstract*
2. Title: Title: *title*
Abstract: *abstract*
20. Title: Title: *title*
Abstract: *abstract*
New paper:
<Title: *title*
Abstract: *abstract*
26


Reviews:
[{"review_id": "FMR5MZC8me", "reviewer": "Reviewer_1agp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is well-written. I especially appreciate the content describing how the reviewer-matching algorithm works. This information is useful to readers who do not have any background knowledge on how these reviewer-matching algorithms work.\n\n(2) The details on the dataset seem sufficient.\n\nWeaknesses:\n\n(1) The main weakness that the authors have repeatedly pointed out as a limitation is the small size of the dataset and lack of diversity. The authors acknowledge this as a limitation. \n\n(2) The section on evaluation of expertise (Section 4.3) is unclear to me. Where did the authors obtain this expertise score? Do we have expertise scores for all paper-author pairs or only the pairs where the paper was selected by the author in the survey? On a related note, I think the data collection process is a bit whacky. Each survey respondent is asked to contribute 5-10 papers that they think they are an expert in. These papers are then aggregated across all respondents. However, this is not how conference reviewing works. During a conference, the pool of papers is determined by what the authors submitted, not what the reviewers want to read. For a given reviewer, there may not be any papers in the review pool that the reviewer wants to read. Therefore, there is a disconnect between the way the dataset is collected and real-world reviewer-matching.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes a public dataset for evaluation of similarity prediction algorithms for reviewer-paper assignment. The paper presents results from baseline similarity computation algorithms used by major conferences on the dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest minor revisions to Section 4.3 clarifying the expertise score. Are survey respondents asked to give their expertise score for all 463 papers or just the ones they selected? If so, how do you handle the paper-reviewer pairs where there is no expertise score?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZGvrqCjHpR", "reviewer": "Reviewer_1GxW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weaknesses:**\n\n*Algorithm Evaluation Setup:* \n* The evaluation approach used by the authors may not capture the full picture, may give suboptimal matches advantage and may not give enough coverage for papers. For example, pairing a reviewer with only two papers may not reflect a more realistic situation where a paper needs to be matched with the best set of reviewers. Instead, starting with pairing a paper with a set of researchers to find the top N best fits may be a better approach. The authors could benefit from better explaining their rationale behind this evaluation setup (at least why only 2 papers+1 resarcher but not considering 2 researcher+1 paper). \n\n* Additional metrics can be added such as as MRR, NDCG, P@k. \n\n* How to handle missing data for evaluation? \n\n*Data Biases:*\n* There are several unintended potential biases in the dataset such as the size and demographic of participants, institutional affiliation, publication history. To be fair, the authors address most of these limitations, but the paper would benefit from a more detailed discussion and resolution of some of these of how these factors might influence the results, extending the dataset if possible, especially when the data is primarily aggregated by mean scores to check statistics. \n\n* Additionally, a minor issue: it would be useful if authors could confirm whether papers newer than the researcher's publication history were excluded from the dataset.\n\n* Also, ties might be a problem for a larger dataset.\n\n*Data preparation*: \n* It would be helpful if the authors provided more details on the dataset, such as the number of papers exists in the arxiv and some key statistics about the content. The inclusion of arXiv identifiers is a good start, but for long-term access, the authors might consider providing additional backup resources (e.g., cleaned up text versions) in case any papers are removed from arXiv. This would ensure ease of access for users. (for future attempts to use the content)\n\nStrengths:\n\n*Importance to the field:*\n* This dataset is a valuable contribution to the reviewer-paper similarity field, mainly due to its public availability and labeling. Its openness allows other researchers to build on this work and reproducibility in the field. \n\n*Dataset description and details:* \n\n* The authors provide a solid description of the dataset and visualizing the expertise counts. More information could be visualized similarly, such as the distribution of differences for the triplets, could offer additional insights into the dataset's characteristics.\niewer-paper pairs) could offer additional insights into the dataset’s characteristics and the challenges of reviewer-paper matching.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the task of author-paper similarity and matching, aiming to improve reproducibility in this area. They release a public dataset of 477 expertise scores from 58 researchers,where researchers themselves provided these scores, with scores ranging from 1 to 5 (with 0.25 increments).  The authors pair papers for each reviewer to assess the performance of various algorithms such as a basic baseline which predicts 1 all the time, TPMS, Elmo, Specter, Specter+MFR algorithm and ACL’s algorithm.. The evaluation metric used is a score-weighted error based on Kendall’s Tau distance. Their findings reveal that current algorithms have high error rates, ranging from 12% to 43%, with Specter+MFR performing the best. The paper also discusses limitations of the dataset, such as its size and distribution, and suggests potential improvements such as expanding the dataset. Regarding algorithms, the authors mentioned the necessity of fine-tuning models before evaluation, and noted that many existing models do not fully utilize available data, such as full-text papers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* This dataset really benefit from if the authors increase the dataset size, as they have mentioned the need for it. Additionally, including more reviewers from different institutions, as well as those with fewer and more papers, would further enhance the diversity and applicability of the dataset.\n\n* Including paper-centric triplets and other ranking metrics in the evaluation could improve the evaluation mechanism by providing a more comprehensive view of the algorithm's performance.", "broader_impact_concerns": "The authors already mentioned potential biases and limitations in a separate section called Limitations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9uCrdSwLBF", "reviewer": "Reviewer_GaY4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) Largely well-written and clear;\n2) the problem is very important and relevant, especially as we try to scale to larger and larger number of submissions. This paper should lead to increased interest in the community to work on this problem;\n3) I really appreciate the authors discuss the limitations (sampling biases) of the dataset;\n4) the empirical evaluations are useful and relevant, mostly with minor gripes. \n5) related works on similarity score calculations and paper matching are well-discussed. \n\nLimitations:\n1) The authors have already talked about sampling biases. There is sampling bias in terms of over-representation from a field, as well as from a particular geographical region. There is additional sampling bias from self-reporting which may not be representative of the true underlying population. \n\n2) No discussion of time taken to match the papers by various algorithms, and how scalable that is. \n\n3) the centarl contribution of the paper is the dataset and the mechanism to create it, but there is no discussion on the survey design background and related works. This is not a CS problem though, may be more along market research or anthropology (I dont know). \n\n4) The empirical evaluation could certainly be expanded, but like the authors said this is not their main contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The central contribution is that the authors provide a new dataset, and a survey mechanism to add to it, for the crucial paper matching problem. There are smaller contributions about evaluations on the said dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can the authors please add a discussion and/or graphs for time taken for each method?\n\nI would suggest the authors discuss the survey and sampling biases with someone who is an expert in the field. I am unable to comment on how well-designed the survey is, and how to tackle various biases. To start the c onversation of this problem, this could be listed as a future work to encourage more people to participate in these surveys. \n\nWould it be possible to add an experiment on the \"top-k\" matches for the loss functions? Akin to NDCG@k instead of NDCG, so it'll be kendall tau up till k ranks, where lower matching scores are sort of ignored because they are not relevant.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, the reviewers were happy with the paper, with all recommending to accept.  There were several questions raised during the reviews.  For instance, there were multiple questions raised about data set size, bias, and evaluation metrics.  These issues were raised during the rebuttal and revision, and the reviewers indicated that they are happy with the current version.  Reviewers also noted that the paper is well-written and the problem is interesting and relevant.
Decision: Accept (poster)