An Analysis of Tasks and Datasets in Peer Reviewing (2024)

Abstract: Taking note of the current challenges of the peer review system, this paper inventories the research tasks for analysing and possibly automating parts of the reviewing process, like matching submissions with a reviewer’s domain of expertise. For each of these tasks we list their associated datasets, analysing their quality in terms of available documentation of creation and use. Building up on this, we give a set of recommendations to take into account when collecting and releasing data.

heading: 1 Introduction
text: Peer Reviewing is a vital part of academic integrity since as early as 1665, when Henry Oldenburg founded the Philosophical Transactions of the Royal Society and established the concept of reviewing the work of others (Kachooei and Ebrahimzadeh, 2022). Since then, the concept of peer review has remained largely unchanged, and nearly every journal has adopted it to select the best publications to publish. In recent decades, peer reviewing has become more of a burden for various reasons, like the overwhelming number of article submissions to review, their often less than optimal quality of content, lack of time to keep up with the majority of published research, or the fact that reviewing is voluntary and researchers find they have less time to review other work.
Technological advances that allow to share research and cooperate worldwide combined with the increasing number of researchers have caused a dramatic increase in the research published daily. The availability of Large Language Models (LLMs) has in fact simplified some tasks of the scientific writing process, like proof reading or text reformulation. This means that the time from idea to written research is significantly shorter. In the Computer Science domain, the number of publications over the last 10 years (2014–2023) has increased dramat-
ically, submissions to ACL have increased sevenfold, and submissions to NeurIPS eight-fold1.
Accounting for these factors, it is harder to find sufficient numbers of reviewers for such conferences. In 2016, about 33% of the NeurIPS reviewers were PhD students (Shah et al., 2018), a percentage that is most likely higher today. The outcome is that there are more inexperienced researchers that have to review larger numbers of submissions, eventually leading to reviews of lower quality.
In recent years, there are efforts to automate and optimize the different peer review steps to lighten the load of the reviewers, area chairs, and programme committees2. Researchers interested in this domain have defined research tasks and collected data to address them.
In this work, we make an inventory of tasks and datasets associated with Peer Review Automation and Optimization, and give recommendations on how to collect and publish such datasets. Without claiming that our work is exhaustive, we see it as a necessary contribution to this field. We examine the available datasets, the tasks they have been used or created for, discussing use cases and automation tasks.This work complements the very recently published Dagstuhl Proceedings (Kuznetsov et al., 2024), and the analyses in Drori and Te’eni (2024) or Lin et al. (2023a).
The paper is structured as follows: Section 2 gives an account of the tasks and their datasets in the Peer Review Analysis domain. Section 3 shows the overlaps between various datasets. Section 4 discusses some recommendations on the creation of datasets in the Peer Review domain. Section 5 comments on the current advances in this domain, concluding with Section 6.
1https://github.com/lixin4ever/ Conference-Acceptance-Rate, (Access: May 2024)
2A description of these roles is available at https:// aclrollingreview.org/reviewing
257

heading: 2 Datasets and Tasks
text: In our effort to understand the research challenges in the scientific peer reviewing domain we analyzed 53 datasets that contain peer reviews. For each of them, mostly created in the last 5 years, we looked at the research tasks for which these sets were used.
Datasets: Of these datasets, 37 cover publications that are in the ML and NLP scientific domains, obtained from the OpenReview portal3, 6 are in the NLP domain and obtained from ACL Rolling Review4, 5 datasets cover multiple domains and journals, 3 obtained from F10005 and 2 from PeerJ6. In Tables 1 to 8 we use the following abbreviations to indicate the source of the collected reviews: OR (OpenReview), ACL (ACL Rolling Review), CO (CoNLL and COLING), F (F1000 Journal), PJ (PeerJ), RS (ShitMyReviewerSays), and SG for self gathered data.
Though most of the datasets can be used for more than one task, in our work we choose to assign them to the task they were used for at their creation time. What we observed is that most of the researchers in this domain opt to create their own datasets from scratch using the OpenReview API, instead of using one of the many available datasets. This affects the reproducibility and comparability of research results, as not all datasets are publicly available or well documented to foster their reuse.
Tasks: Reviews can be looked at with different research aims in mind. Earlier work (Cho, 2008), for example, analyzed the helpfulness of student reviews, while more recent work looks at various aspects of peer reviewing, like Discourse Analysis (Kennard et al., 2022; Ruggeri et al., 2023), Peer Review Quality (Verma et al., 2023; Ghosal et al., 2022b; Bharti et al., 2023) or meta review generation (Chen et al., 2023; Ridenour et al., 2022).
In the following we will discuss the most common tasks and datasets for peer reviewing.

heading: 2.1 Review Analysis
text: Tasks grouped under this title aim to understand how reviews impact the submissions they were written for, aiming to detect biases in scoring, for example. For this area we identified three datasets (Table 1). The first one listed is used to look at
3https://openreview.net/ 4https://aclrollingreview.org 5https://f1000research.com/ 6https://peerj.com/
how review scores for submissions to ICLR 2017- 2022 affect their acceptance rate and their future citations (Wang et al., 2023). The second listed dataset was used in research that looked at peer reviews for journal submissions in the medical domain (BMC, BMJ, PLOS Medicine), examining their length and quality (Geldsetzer et al., 2023). The “ICLR Database” (Zhang et al., 2022) has been used in research that analysed fairness disparities in the review process and the geographical and institutional distribution of submissions.

heading: 2.2 Reviewer Assignment
text: In content preparatory phases of a conference, submissions are assigned to reviewers, ideally, based on their experience and interests. Most often, this is done via a bidding process, after which reviewers get assigned papers to review, according to their bids. This bidding process is, nevertheless, prone to manipulation, such as reviewers either voting to review submissions from their friends and close colleagues (choosing to grade them highly) or, when having sent in submissions, bidding to review similar submissions to their own, then negatively scoring them to improve their own submission acceptance chances, as the dataset (Jecmen et al., 2023) and research in Jecmen et al. (2024) demonstrate. To avoid this kind of malicious bidding, one possibility is to automatically match submissions to reviewers. Two further datasets created to investigate the effectiveness of such matching and, at the same time, to analyse the impact of malicious bidding are listed in Table 2.
The dataset and the method introduced by Stelmakh et al. (2020) detect malicious behavior at the meta-review level. To evaluate different methods of fair assignment of reviewers, Stelmakh et al. (2023) created a gold-level dataset on the self-reported expertise of reviewers for given publications.

heading: 2.3 Score and Acceptance Prediction
text: To ease decision making, reviewers are asked to numerically score the different aspects of a submission, such as Soundness, Presentation, Contribution, and give an Overall Score. These scores are then used by area chairs to decide on the acceptance of a submission, and to formulate their meta-review. Fernandes and Vaz-de Melo (2023, 2022) analyzed the Review Score Prediction (RSP) and the Paper Decision Prediction (PDP) tasks, by evaluating models on two self-mined datasets of ICLR conference submissions (Table 3). The two datasets mainly introduced in their research differ in the number of reviews (14,459 vs 18,170) in them. In their work, they trained models on reviews, to predict RSP and PDP values on Submission-Review tuples. This was similar to the dataset and prediction task, proposed by Kang et al. (2018), who created one of the first Peer Review datasets, to allow score prediction as well as review generation.
Liu et al. (2023b) conducted a study on the willingness of reviewers to change their scores after the author rebuttal and revision phases. Specifically, they conducted a randomized controlled trial on 108 participants to find under which conditions reviewers tend to change their original scores. With PEERAssist, Bharti et al. (2021) explored score prediction with a deep neural architecture, and outperformed a similar architecture using sentiment analysis (Ghosal et al., 2019). Ribeiro et al. (2021) used a private dataset of two non-disclosed conferences to perform RSP, PDP, and sentiment analysis to extract review polarities. Datasets that tackle these tasks are listed in Table 3.

heading: 2.4 Revision, Dialogue, Rebuttal Analysis
text: Some peer review processes include a dialogue between the authors and the reviewers, a dialogue with visible effects on the final version of a publication. Analyzing differences between consequent submission versions as well as the dialogue among reviewers or between reviewers and authors is, therefore, of interest. Having access to such information allow experiments on predicting changes
over time or finding disagreements between different parties. Table 4 lists the peer review datasets used for approaching any of these tasks.
Among the datasets that investigate the differences between submission versions we mention CASIMIR (Jourdan et al., 2024), ARIES (D’Arcy et al., 2023), and arXivEdits (Jiang et al., 2022). The CASIMIR collection contains 3.7 million sentence pairs, from different submission revisions. A possible use case is the analysis of differences between revisions and prediction necessary revisions. The ARIES dataset has been created for the task where revised submissions are annotated to trace the edits back to the reviewer comments. arXivEdits does not contain reviews, but a set of 751 arXiv publications with revision history and sentence alignment across multiple paper revisions.
Datasets were created with the aim to understand and classify the arguments and disagreements made in the reviews, like APE (Argument Pair Extraction) (Cheng et al., 2020) and the AMPERE dataset (Hua et al., 2019). The APE dataset contains 4,764 argument pairs from the review-rebuttal phases, while the AMPERE dataset contains proposition and argument type annotations, aiming to answer research questions on argument classification. In the same direction, Singh et al. (2021) analyzed peer reviews looking at discussion comparisons, to better understand the factors that affect the acceptance decisions. Another dataset that explores disagreements between reviewer comments is ContraSciView (Kumar et al., 2023), a dataset of around 28,000 annotated review pairs, if they agree or contradict each other. They further proposed a model to detect contradictory statements in Peer Reviews. Finally, here, we note the ArgSciChat dataset (Ruggeri et al., 2023) containing argumentative dialogues, argumentative and exploratory questions and answers for 20 papers in the NLP domain.
Rebuttal strategies are of interest to researchers, with data collected for and work looking into rebuttal generation based on reviewers’ comments and evaluations (the Jiu-Jitsu dataset (Purkayastha et al., 2023)). Huang et al. (2023) conducted an empirical study on ICLR papers, to detect the most common rebuttal strategies and predict the changes in review scores before and after the rebuttal phase. Similarly, DISAPERE (DIscourse Structure in Academic PEer REview) (Kennard et al., 2022) is a dataset of 20,000 labeled review-rebuttal sentences
combinations, to analyze the interpretation of reviews and response strategies. ReAct (Choudhary et al., 2021) is a dataset that falls into this group (rebuttal and revision strategy support). The authors annotated review sentences such that, on one side, review recommendation can be quickly classified into actionable (possible to answer in a revision), and on the other side the revision action can be classified by finer-grained types, like Suggestion, Question, or Disagreement.

heading: 2.5 Review Generation
text: While review generation may solve some of the reviewing issues, such as speeding up the academic review process or generating high quality reviews, it may also introduce new threats into the academic review process. For example, authors could tune their submissions so that automated review models would accept them.
Table 5 shows a list of datasets suitable for review generation tasks. The first model for automatic review generation is ReviewRobot (Wang et al., 2020), trained on data from OpenReview combined with multiple knowledge graphs to obtain domain knowledge to conduct reviews. In 2022 Yuan et al. (2022) published their ASAPReview dataset and a model for review generation, but found that their model is less constructive and less factual then human-written reviews. Another dataset published for researchers to work on the generation of reviews and the prediction of scoring results is the ORB dataset (Szumega et al., 2023). The dataset is, though, not well documented.
With every new LLM release, researchers make use of them to generate reviews. One dataset to enable their use is, for example, the ReviewRevision Multiple-Choice Questions (RR-MCQ) dataset, that allows to have a quick assessment on
LLM performance in the scientific review domain (Zhou et al., 2024). Similarly, Liu and Shah (2023) created a dataset to identify errors in computer science publications, where they manually modified the publications to contain factual inaccuracies.
Kuznetsov et al. (2022) looked into assisting the peer review process providing several datasets for tasks like review generation, pragmatic tagging, or score prediction. The Yes-Yes-Yes dataset (Dycke et al., 2022) contains review from the ACL Rolling Review, 2021, and, later, the same authors created F1000RD (Kuznetsov et al., 2022), the first multidomain corpus of peer reviews of the F1000Research platform. These datasets were combined and released later (Dycke et al., 2023a). Based on the F1000RD dataset, a Pragmatic Tagging dataset was released for the PragTag SharedTask (Dycke et al., 2023b).

heading: 2.6 Metareview Generation
text: Writing metareviews is a task performed by Area Chairs, who read through all the reviews for a submission, check on disagreements between them, and provide a summary of the reviews with a final decision. Table 6 lists the datasets used in research on metareview analysis and generation.
Sun et al. (2024b) created an own peer review dataset, covering five years of review data from OpenReview, and trained a model on it to generate metareview texts. PeerSum (Li et al., 2023) is a different dataset for metareview generation, which features explicit conflicting information in source documents that the meta-reviewers have to handle. ORSUM (Zeng et al., 2023) is a corpus for scientific opinion summarization, which uses a undisclosed dataset from 39 conferences and workshops that published their peer reviews on OpenReview. The MOPRD (Multidsciplinary Open Peer Review Dataset) (Lin et al., 2023b) dataset was created to enable metareview generation, editorial decision prediction, author rebuttal generation, and scientometric analysis of papers from multiple disciplines.
It is composed of paper metadata, manuscripts of the initial submission and multiple revisions, review comments, meta-reviews, author rebuttal letters, and editorial decisions of papers from Biology, Chemistry, Computer Science, Environment and Medicine published on PeerJ. Wu et al. (2022) constructed a Peer Review and Rebuttal CounterArguments (PRRCA) dataset, which contains both the reviews, and the authors responses to model the peer review evaluation process.
The Meta-Review Dataset (MReD) (Shen et al., 2022) is a sentence-level annotated dataset of 23,675 reviews and their corresponding 7,089 metareviews. Each sentence is categorized into one of nine categories (abstract, strength, weakness, rating summary, area chair disagreement, rebuttal process, suggestion, decision, and miscellaneous) to then generate the meta-reviews based on these sentences.

heading: 2.7 Quality Analysis
text: Analyzing reviews in scientific papers can help to measure the helpfulness of the conducted peer review, the politeness or harshness of the peer reviewer, the reviewer confidence on a given topic, or the reviewer disagreement. Research in this area aims to detect and measure quality issues in conducted reviews. Table 7 presents a list of datasets for quality analysis of peer reviews.
The datasets in this subsection are used in, roughly seen, two types of tasks. One is classifying the reviews by detecting politeness levels and harshness (e.g., PolitePEER in Bharti et al. (2023) or Verma et al. (2022)), and detecting review polarities (Wang and Wan, 2018). The other task is exploring the content of the reviews from different argumentative points of view, in order to detect disagreement between reviewers and scoring differences between reviewers and submission versions (Chakraborty et al., 2020; Gao et al., 2019), or to estimate the review helpfulness by checking its thoroughness (Severin et al., 2023) and detecting suggestions for improvement to the submission authors (Pfütze et al., 2022), finding arguments
in peer reviews as in AMSR (Argument Mining in Scientific Reviews) (Fromm et al., 2021).
A combination of the two types of tasks is seen in ReviVal (Verma et al., 2023), in which the authors use the dataset to grade reviews based on Exhaustiveness and Strength of the review texts, or in SubstanReview (Guo et al., 2023), in which the authors look at substantiation levels of reviews by verifying that subjective statements (claims) are backed up by justifications (evidence). HedgePeer (Ghosal et al., 2022b) provides a dataset for uncertainty detection in peer reviews by annotating the review comments in terms of hedge cues and spans. Finally, the dataset introduced in by Ghosal et al. (2022a) aims to facilitate the analysis of the reviewers’ confidence on certain sections and aspects of the paper and then use those insights to investigate further the quality of peer reviews.

heading: 2.8 Citation Prediction
text: An interesting task, which is not directly associated with the Peer Review domain, is to predict the popularity and citations of accepted publications based on their reviews. To our knowledge, there currently are two datasets on this topic (see Table 8). The CiteTracked (Plank and Dalen, 2019) dataset links reviews from the NeurIPS conference with their citations to predict the impact of the submissions. Li et al. (2019) performed a similar study on ICLR reviews, but provided further data and metadata to their models, such as information about the authors.

heading: 3 Dataset Similarities and Discrepancies
text: In the last years, OpenReview has become the primary data source for review-related tasks, and, therefore, many of the published datasets overlap. We look more closely into the datasets which, among others, contain the ICLR conference review data (35 sets, see Table 9).
We detected several inconsistencies across these datasets coming from the same primary source (here, OpenReview). One of them is in the number of reviews for one year of the conference (2017). Namely, Kang et al. (2018); Verma et al. (2023) report 427 submissions, while Wang and Wan (2018); Zhang et al. (2022) and Chakraborty et al. (2020) reported 490 and 491 submissions (the latter difference possibly being a reporting error on the workshop track). Later, Bharti et al. (2021) cleaned this dataset by removing duplicates or empty reviews and reports 354 qualitative reviews for ICLR 2017. These differences can be due to authors agreeing to publish their submissions later than the dataset creation time, not filtering out duplicates, including workshop submissions7, or, more likely, are due to reporting inaccuracies. We found, for example, that Zhang et al. (2022) reported 47 workshop papers. Some studies excluded data due to poor quality or availability issues (Fernandes and Vaz-de Melo, 2023; Cheng et al., 2020; Chakraborty et al., 2020), while other studies avoid any kind of reporting on data quality and filtering.
Most of the datasets were extended with annotations to allow the evaluation of the conducted experiments. However, the number of annotations is rather small (in the hundreds), and do not follow a common structure, to allow the integration of multiple datasets and the interchangeability of datasets. While many of the datasets contain several hundred annotated sentences or reviews, such as D’Arcy et al. (2023) and Zhou et al. (2024); others, like Kumar et al. (2023) and Jourdan et al. (2024) annotated larger amounts of data. The ASAP-Review dataset was extended by manually annotating 8,582 publications with sentence level labels on sentiment and aspect categories (Kumar et al., 2023; Yuan et al., 2022). Jourdan et al. (2024) automatically aligned more than 3.7 million sentence pairs for revision analysis and labeled the most likely intention behind each edit.
7Workshop submissions are different from the main conference submissions as they are often reviewed more leniently than conference submissions.
Furthermore, different conferences use different review guidelines and numerical scoring scales, which can lead to biases if these conferences are combined. For example, ICTIR allows reviewers to score papers on a scale from -3 to 3, while NeurIPS uses a scale from 1 to 10.

heading: 4 Guidelines for Dataset Publication
text: As is obvious from the previous sections, the number of datasets recently published in the area of scientific peer review analysis is already considerable, although their re-use is impeded by the nontransparent way in which the data was collected. Therefore, we advocate for the adherence to data creation and publication guidelines such that their reuse is facilitated. Making sure that such datasets are findable, that they are well documented, and additionally, that machine-readable meta-data about them is available are ways to contribute towards their reuse. The current list of available datasets published following the above-mentioned Dagstuhl seminar8 is a first step toward achieving this, however incomplete or lacking machine-interpretable data and meta-data. Some of the recommendations we make in the following subsections are, in fact, aligning with the FAIR principles for data sharing9.
Furthermore, whenever possible, we recommend researchers to describe their datasets through appropriate metadata schema10.

heading: 4.1 Dataset Reuse
text: As discussed in the previous section, there are many overlaps between existing datasets. Although existing datasets may not fit all research questions, there are several datasets targeting the same task that overlap significantly (Shen et al., 2022; Zeng et al., 2023; Sun et al., 2024b). Therefore, an obvious recommendation is to investigate existing datasets for their appropriateness to the task at hand. Ideally, such datasets are visible to search portals like DataCite11 or OpenAire12. Overview studies like our work or Kuznetsov et al. (2024) are for now an alternative. If necessary these datasets should be extended instead of creating a new dataset from similar underlying sources.
8https://github.com/OAfzal/ nlp-for-peer-review
9FAIR: Findable, Accessible, Interoperable, Re-usable https://www.gofair.foundation/interpretation
10E.g. schema.org/docs/data-and-datasets.html 11https://datacite.org/ 12https://www.openaire.eu/

heading: 4.2 Preprocessing and Annotation
text: When data collected from the open review portals is processed or annotated, details about the processing should be made available. This includes annotation guidelines, inclusion or exclusion criteria for the data points in the final dataset, software. This type of documentation is relevant to researchers who need to understand how data was obtained in order to decide its appropriateness to their task. We found little evidence of preprocessing and annotation documentation. This issue was confirmed, for example by Bharti et al. (2021), who mentioned the
lack of documented preprocessing on the datasets when reporting on duplicates and empty reviews in the ICLR 2017 data.

heading: 4.3 Availability and Documentation
text: A key factor for dataset reuse is that it is actually available and the data is provided in an easy to process form, such that integrating it into other processing pipelines needs little overhead.
Many of the datasets that we refer to are available for download, for the remaining, however, either only data acquisition scripts (Geldsetzer et al., 2023) or insufficient selection criteria and indica-
tion of review platform source are provided (Matsui et al., 2021; Lin et al., 2023b; Liu and Shah, 2023; Sun et al., 2024b). In some cases we have found outdated links (Li et al., 2023).
Besides data availability, another issue is the missing documentation of the datasets contents. Several of the published datasets state that they investigated publications from the OpenReview platform, but do not elaborate on which conference and years they use (Bharti et al., 2023; Zeng et al., 2023; D’Arcy et al., 2023; Szumega et al., 2023; Jourdan et al., 2024), making that research difficult to reproduce, if at all possible.
Therefore, we recommend clearly stating the selection criteria for the data (e.g. conferences, years, platforms), whether the data is built on previous datasets (e.g., Guo et al. (2023)), and report descriptive statistics on the released collection, not only for the full dataset, but also for slices of the data (see Sun et al. (2024b)).

heading: 4.4 Canonical Splits
text: The datasets we reported on can be, and are, used both for analysis and for training machine learning models. To make the results of classification or generation tasks comparable, in the classical ML tradition, such datasets should have predefined canonical splits for training, validation, and testing. PeerRead (Kang et al., 2018), ORSUM (Zeng et al., 2023), PolitePeer (Bharti et al., 2023), SubstanReview (Yuan et al., 2022) do contain such predefined splits, but others do not (e.g. Bharti et al. (2021); Kumar et al. (2023); Purkayastha et al. (2023)).
Thus, we recommend to define canonical data splits to make model performance comparable.

heading: 4.5 Licensing
text: As already stated in, for example, (Kuznetsov et al., 2022; Dycke et al., 2022, 2023a) many of the datasets here are not licensed and ethically questionable, as reviewers may not have the option to opt-out of their reviews being published. Furthermore, many of the datasets are published without a clear license, although the original data is licensed. An example of this is the OpenReview platform, which is licensed as CC-BY 4.013, but the datasets built on it rarely mention or re-apply this license.
Thus, we advise using clear licenses for published data, and pro-active data collection whenever possible, as proposed by (Dycke et al., 2022).
13https://openreview.net/legal/terms

heading: 5 Discussion
text: The recent Dagstuhl Proceedings, What Can Natural Language Processing Do for Peer Review, suggest that it is mostly up to researchers in the NLP domain to solve many tasks until AI-assisted Peer Reviewing or Automated (Peer) Reviewing becomes possible. While this is true for most of the tasks in Section 2, there are challenges that, we believe, will profit from solutions that don’t rely solely on Large Language Models or Text Analysis.
In our work on this paper, we note that research in peer review (Liu and Shah, 2023; D’Arcy et al., 2024; Ghosal et al., 2019; Fytas et al., 2021; Sun et al., 2024a) aim to automatically generate or predict peer reviews, leaving out the use of domain knowledge to measure the impact, novelty or compare the results to previous research results that use same datasets. One can argue that the knowledge is already encoded in LLMs, we comment that its use is not tractable. To our knowledge, ReviewRobot (Wang et al., 2020) is the only tool that uses Controlled Knowledge-Driven-Generation, to generate reviews using domain knowledge by accessing multiple knowledge graphs.The quality of the results is, however, heavily dependent on the quality of the information extraction processes that provide the content of the knowledge graphs.
In the IR community, information extraction is a long-standing research task, the latest efforts for fine-grained entity extraction being visible in the organization of the RUFEERS task at TREC14, and the State-of-the-Art task at CLEF15. We hope that research in information extraction may contribute to automated population of knowledge graphs, noting that the most extensive knowledge graph (Open Research Knowledge Graph) (Kabongo et al., 2024), which supports comparing scientific publications by various parameters (including datasets), is currently curated by hand.
In conclusion, we believe connecting the research on LLMs with Information Extraction as pursued by the IR community or the Semantic Web community, can help extract the contributions of papers and compare their results to previous ones and effectively support reviewers in their work.

heading: 6 Conclusion
text: In this work, we gave an overview of 53 different datasets for scientific peer reviewing and the tasks
14https://tac.nist.gov/2024/RUFEERS/ 15https://simpletext-project.com/2024/en/
these datasets have been created for. We found several issues with these datasets, such as poor documentation, overlapping data, unclear licensing, and a lack of canonical splits. For this reasons we propose several guidelines for publishing new datasets in this domain, guidelines that, in fact, apply to publishing datasets in general. We also see a need for the creation of a taxonomy of tasks in AI-assisted or automated peer reviewing, and the need for a well-documented dataset that consists of peer reviews from various domains, which can be used for different subtasks.
Currently, the analysis of peer review texts and tool or model development has been done by researchers in the NLP domain. We hope that researchers in other Computer Science fields will assist with their expertise. The Semantic Web community or the Information Retrieval community, are currently only slightly involved in this process. Therefore, we believe that it is necessary to get these communities more involved in this research. This can be done via future shared-tasks, and, although we see the irony, new datasets and tasks that are tailored to make use of the expertise in these domains.

heading: Acknowledgments
text: While this paper critiques the numerous datasets available, we recognize that creating a dataset is an extremely challenging task. Therefore, as a community, we should be thankful and acknowledge the efforts of everyone who creates and shares datasets with the community.

heading: Future Work
text: This can be done via future shared-tasks, and, although we see the irony, new datasets and tasks that are tailored to make use of the expertise in these domains