Venue: NAACL 2022

# Disapere: A Dataset for Discourse Structure in Peer Review Discussions

Neha Nayak Kennard Tim O'Gorman Rajarshi Das Akshay Sharma Chhandak Bagchi Matthew Clinton Pranay Kumar Yelugam Hamed Zamani Andrew McCallum University of Massachusetts Amherst {kennard, togorman,rajarshi, akshaysharma, cbagchi, mfclinton, pyelugam, zamani, mccallum}@cs.umass.edu

###### Abstract

At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined.

We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors' stance towards review arguments. Further, we annotate _every_ review and rebuttal sentence.

We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1234 - 1249

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Peer review performs the essential role of quality control in the dissemination of scientific knowledge. The recent rapid increase in academic output places an immense burden on decision makers such as area chairs and editors, as their decisions must take into account not only extensive manuscripts, but enormous additional amounts of technical text including reviews, rebuttals, and other discussions.

One long term goal of research in peer review is to support decision makers in managing their workload by providing tools to help them efficiently absorb the discussions they must read. While machine learning should not be used to produce condensed accounts of the peer review text due to the risk of amplifying biases Zhao et al. (2017), ML tools could nevertheless help manage information overload by identifying patterns in the data, such as argumentative strategies, goals, and intentions.

Any such research requires an extensive labeled dataset. While the OpenReview platform Soergel et al. (2013) has made it easy to obtain unlabeled public peer review text, labeling this data for supervised NLP requires highly qualified annotators. Correct interpretation of the discourse structure of the text requires an understanding of the technical content, precluding the use of standard crowdsourcing techniques. Prior work on discourse in peer review has focused this qualified labor force on labeling arguments extracted from the text, which enables the complete annotation of more examples, at the expense of research on non-argumentative behaviors in peer review. While there has been extensive research and deep analysis of different aspects of peer review, the taxonomies used to describe review argumentation are disparate and not directly compatible. Finally, there has been limited research into understanding the discourse relations between rebuttals and reviews Cheng et al. (2020); Bao et al. (2021), and none so far into the discourse structure of rebuttals.

This paper presents **DISAPERE** (**DIS**course **S**tructure in **A**cademic **PE**er **R**eview), a dataset focusing on the interaction between reviewer and author1. We give reviews and rebuttals equal importance, and emphasize the relations between them. To enable the study of behaviors beyond the core arguments, we also annotate every sentence of both the review and rebuttal, and provide fine-grained labels for non-argumentative types. We annotate at the sentence level not only for completeness but also to avoid the propagation of errors from argument detection. We annotate four properties (review-action, fine-reviewed, aspect, polarity) of each review sentence, where the set of properties and their values were developed by synthesizing taxonomies from

prior work. We also annotate each sentence of a rebuttal with a fine-grained label indicating the author's intentions and commitment, and a link to the set of review sentences that form its context. Figure 1 shows the DISAPERE annotation scheme on a minimal, fictional example review-rebuttal pair.

DISAPERE is intended as a comprehensive and high-quality test collection, along with training data to fine-tune models. Our annotations are carried out by graduate students in computer science who have undergone training and calibration, amounting to over 850 person-hours of annotation work. Much of the test data is double-annotated, and we report inter-annotator agreement on all aspects of the annotation. We describe the performance of state of the art models on the tasks of predicting labels and contexts, showing that interesting ambiguities in the data provide the NLP community with research challenges. We also show an example that demonstrates how decision makers could use models like these to understand trends and inform policies for future conferences (SS 5).

The contributions of this paper are as follows: (1) a new labeled training dataset of 506 review-rebuttal pairs (over 20k sentences) of peer review discussion text in English, where review sentences are annotated with four properties, and rebuttal sentences are annotated with context and labels from a novel scheme to describe discourse structure; (2) a taxonomy of discourse labels synthesizing prior work on discourse in peer review and extending it to add useful subcategories; (3) a summary of the performance of baseline models on the dataset (SS 6); (4) examples of analyses on the dataset that could benefit peer review decision makers (SSSS 4 and 5), and (5) extensive annotation guidelines and software to support future labeling efforts.

## 2 Related work

The design of this dataset draws upon extensive, but disparate prior work on this topic. Many works, some addressed below, have taken advantage of the availability of review text hosted on OpenReview.

Argument-level review labelingPrior work has developed label sets that address different phenomena. Hua et al. (2019) introduced the study of discourse structure in peer review by annotating argumentative propositions in the AMPERE dataset with a set of labels tailored to the peer review domain (evaluation, request, fact, reference, and quote). Similarly, Fromm et al. (2020)'s AMSR dataset frames the problem as an argumentation process, in which the stance of each argument towards the paper's acceptance or rejection is of paramount importance. Both view peer review as argumentation, using argument mining techniques to highlight spans of interest.

While its goal is not to examine discourse structure per se, Yuan et al. (2021) uses polarity labels to indicate each argument's support or attack of the authors' bid for acceptance. Besides polarity, these examples follow Chakraborty et al. (2020) by annotating each argument with the _aspect_ of the paper it comments on.2 In contrast to Yuan et al. (2021), we do not attempt or recommend generat

Figure 1: A depiction of our annotation scheme on a minimal, fictional review-rebuttal pair. A: Review-action , including Structuring, Request, Evaluation; B: fine-review-action , fine-grained categorization of Structuring and Request sentences; C: aspect , indicating the qualities of the manuscript being commented upon D: polarity indicating whether these comments are positive or negative in nature. E: Each sentence in the rebuttal is mapped to zero or more sentences in the review, which constitute its context. This is a many-to-many relation. F: The sentences in the rebuttal are labeled with domain-specific discourse acts (rebuttal-action ); each discourse act may be categorized according to whether it concurs with (✓) or disputes (\(\times\)) the premise of the context it is responding to.

 

ing peer review text, instead focusing on _analyzing_ human-generated text in peer review.

Review-rebuttal interactionsWe also expand on work by Cheng et al. (2020), who first annotated discourse relations between sentences in reviews and rebuttals. While Cheng et al. (2020, 2021) present new deep learning architectures, in this paper we focus on the creation and comprehensive annotation of a new dataset, illustrated with results from some less specialized baseline models.

Other research into rebuttals includes Gao et al. (2019). Besides their main finding that reviewers rarely change their rating in response to rebuttals, they find that more specific, convincing and explicit responses are more likely to elicit a score change. Observations from this paper are formalized into rebuttal action labels in DISAPERE.

Comparison of datasetsIn DISAPERE we attempted to unify these schemas to form a single hierarchical schema for review discourse structure. We then expanded this hierarchical schema to introduce fine-grained classes for implicit and explicit requests made by the reviewers. The details of the correspondence between DISAPERE labels and those from prior work are summarized in Appendix A. In contrast to prior work, DISAPERE labels discourse phenomena at the sentence level rather than the argument level. This enables more thorough coverage of the text while avoiding the propagation of errors from machine learning models earlier in the annotation pipeline. While using manually defined discourse units (above or below the sentence level) may more precisely capture some discourse information, a separate pass of discourse segmentation can hinder the use of discourse datasets, as achieving consistent and replicable annotation of argument units is known to be highly challenging Trautmann et al. (2020), and also because few works actually tackle unit segmentation Ajjour et al. (2017).

## 3 Dataset

Each example in DISAPERE consists of a pair of texts: a review and a rebuttal. Labels for reviews and rebuttal sentences are described below. Review sentence labels are summarized in Table 2, and rebuttal sentence labels in Table 3.

### Review sentence labels

#### 3.1.1 Review actions

review-action annotations characterize a sentence's intended function in the review. Annotators label each sentence with one of six coarse-grained sentence types including _evaluative_ and _fact_ sentences, _request_ sentences (including questions, which are requests for information), as well as non-argument types: _social_, and _structuring_ for organization of the text.

#### 3.1.2 Fine-grained review actions

We also extend two of these review actions with subtypes: _structuring_ sentences include headers, quotations, or summarization sentences, and _request_ sentences are subdivided by the nature of the request, distinguishing between clarification of factual information, requests for new experiments,

\begin{table}
\begin{tabular}{l l c c c c}  & Dataset & **Dataset** & **DIGAR** & **DIGAR** & **DIGAR** \\ \hline \# examples & 400 & 77 & 1k & 4.7k & 506 \\ \# labels & 10k & 1.4k & 5.7k & 130k & 46k \\ \end{tabular} \begin{tabular}{c l l l l l}  & Arg. stmts. & ✓ & ✓ & & ✓ & ✓ \\  & Arg. types & ✓ & & & & ✓ \\  & Polarity & & ✓ & ✓ & & ✓ \\  & Aspect & & & ✓ & & ✓ \\  & Non-arg. & & & & & ✓ \\  & All sents. & & & & ✓ \\ \hline \multirow{8}{*}{
\begin{tabular}{c} **Purchal** 

requests for an explanation (e.g. of motivations or claims), requests for edits, and identification of minor typos.

1.3 Aspect and polarity aspect annotations follow the ACL review form Chakraborty et al. (2020); Yuan et al. (2021). These distinguish _clarity_, _originality_, _soundness/correctness_, _replicability_, _substance_, _impact/motivation_, and _meaningful comparison_. Following Yuan et al. (2021), arguments with an aspect are also annotated for polarity. We label _positive_ and _negative_ polarities. aspect and polarity are applied to sentences whose review-action value is _evaluative_ or _request_.

### Rebuttal sentence labels

We annotate two properties of each rebuttal sentence: a rebuttal-action label characterizing its intent, and its context in the review in the form of a subset of review sentences.

#### 3.2.1 Rebuttal actions

The 14 rebuttal actions (Table 3) are divided into three rebuttal-stance categories (_concur_, _dispute_, _non-arg_) based on the author's stance towards the reviewer's comments.

(1) _concur_: The author concurs with the premise of the context. This includes answering a question or discussing a requested change that has been made to the manuscript, conceding a criticism in an evaluative sentence. (2) _dispute_: The author disputes the premise of the context. The rebuttal sentence may reject a criticism or request, disagree with an underlying fact or assertion, or mitigate criticism (accepting a criticism while, e.g., arguing it to be offset by other properties). (3) _non-arg_: Encompasses rebuttal actions including _social_ actions (such as thanking reviewers), and _structuring_ labels, for sentences that organize the review.

Responses to _requests_ are further annotated: if the author _concurs_, we record whether the task has been completed by the time of the rebuttal, or promised by the camera ready deadline; if the author _dispute_s, we record whether the task was deemed to be out of scope for the manuscript.

#### 3.2.2 Rebuttal context

We refer to the set of sentences which a rebuttal sentence is responding to as the _context_ of that



sentence, with special labels for when referring to the entire review (_global context_) or the empty set (_no context_). By not mandating a fixed discourse chunking, these annotations may handle situations when some rebuttal sentences respond to large sections of text, and other rebuttal sentences respond to specific sentences within those sections.

### Data Source and Annotation

DISAPERE uses English text from scientific discussions on OpenReview Soergel et al. (2013), which makes peer review reports available for research purposes. We draw review-rebuttal pairs from the International Conference on Learning Representations (ICLR) in 2019 and 2020, resulting in text within the domain of machine learning research. Review-rebuttal pairs are split into train, development and test sets in a 3:1:2 ratio such that all texts associated with any manuscript occur in the same subset. Overall statistics for the dataset are summarized in Table 4.

Authors are able to respond to each ICLR review by adding a comment. Although rebuttals are not formally named, we consider direct replies by the author to the initial review comment to constitute a rebuttal. While multi-turn interactions are possible, we focus on reviews and initial responses, and leave study of extended discussion for future work. The text is separated into sentences using the spaCy Honnibal and Montani (2017) sentence separator.

Annotation was accomplished with a custom annotation tool designed for this task, which is available as part of the code release accompanying DISAPERE. The tool is described in detail in Appendix B. Annotators annotate each sentence of a review, then examine the rebuttal sentences in order, selecting sets of review sentences to form their context. While this linking between sentences does not explicitly align multi-sentence chunks as in pipelined approaches to discourse alignment Cheng et al. (2020), we note that since multiple sentences may be aligned to the same set of sen

\begin{table}
\begin{tabular}{l l c c} \hline \hline  & Train & Dev & Test \\ \hline Num. review-rebuttal pairs & 251 & 88 & 167 \\ Num. manuscripts & 94 & 37 & 57 \\ Num. adjudicated pairs & 0 & 0 & 65 \\ Num. review sentences & 5216 & 1484 & 3246 \\ Num. rebuttal sentences & 5805 & 2015 & 3283 \\ Num. review tokens & 112k & 33k & 70k \\ Num. rebuttal tokens & 131k & 44k & 75k \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistics for the dataset. Where possible, multiple reviews for the same manuscript were annotated. All reviews for any particular manuscript fall within the same train/dev/test split. Adjudiated pairs are those that were annotated by multiple annotators, and had disagreements resolved by an experienced annotator. All test set pairs are double-annotated. While the original sentence boundaries were maintained, tokenization within sentences was carried out using StanzaQi et al. (2020).

\begin{table}
\begin{tabular}{c l l l l l} \hline \hline Category & Label & Description & Reply to & Percentage \\ \hline \multirow{6}{*}{\begin{tabular}{} \end{tabular} } & Answer & Answer a question & Request & 32.76\% \\  & Task has been done & Claim that a requested task has been completed & Request & 8.58\% \\  & Concede criticism & Concede the validity of a negative eval. statement & Evaluative & 2.70\% \\  & Task will be done & Promise a change by camera ready deadline & Request & 2.01\% \\  & Accept for future work & Express approval for a suggestion, but for future work & Request & 1.30\% \\  & Accept praise & Thank reviewer for positive statements & Evaluative & 0.35\% \\ \cline{2-6}  & Reject criticism & Reject the validity of a negative eval. statement & Evaluative & 10.37\% \\  & Mitigate criticism & Mitigate the importance of a negative eval. statement & Evaluative & 2.43\% \\  & Reject request & Reject a request from a reviewer & Request & 1.16\% \\  & Refute question & Reject the validity of a question & Request & 0.95\% \\  & Contradict assertion & Contradict a statement presented as a fact & Fact & 0.86\% \\ \hline \multirow{6}{*}{
\begin{tabular}{} \end{tabular} } & Structuring & Text used to organize sections of the review & - & 17.82\% \\  & Summary & Summary of the rebuttal text & - & 7.94\% \\  & Social & Non-substantive social text & - & 6.71\% \\  & Followup question & Clarification question addressed to the reviewer & - & 0.32\% \\  & Other & All other sentences & - & 3.75\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: A list of the rebuttal sentence labels, their descriptions, and the percentage of rebuttal sentences they apply to. The “Reply to” column shows the review-action types that a particular rebuttal type would canonically reply to. Each rebuttal sentence has exactly one rebuttal-action label, so these percentages add up to 100%.

 

tences in the review, some discourse structure is nevertheless latently implied.

### Agreement

We report Cohen's \(\kappa\)Cohen1960 on the IAA of labeling both review and rebuttals, treating each sentence as a labeling unit (Table 5). The annotators for each example are selected randomly from the pool of 10 annotators. Cohen's \(\kappa\) is calculated for sentences annotated at least twice. Where more than two annotations were produced, we calculate \(\kappa\) between all pairs and normalize by the number of possible pairs. The results show between moderate and substantial chance-corrected agreement between annotators, for both rebuttal-action and rebuttal-stance labels (Appendix D provides details about agreement on context sentences). While these IAA scores do illustrate the noise of the task, note that this is not highly unusual for discourse labeling tasks - e.g. Habernal2017 and Miller2019 both report \(\alpha_{u}\) between 0.4 and 0.5.

## 4 Analysis

### Context types

We separate the different types of rebuttal contexts in terms of the number and relative position of selected review sentences in Table 6, along with the four cases in which the context cannot be described as a subset of review sentences. Notably, 84.81% of sentences are linked to some review context. A small number of sentences refer to other sentences within the rebuttal, rather than any review context, posing a challenge for future work.

### Alignment

One might reasonably hypothesize that the task of alignment between rebuttal and review sentences would be trivial, since authors are likely to respond to each point in the review in order. We can show that this is not the case. In Figure 2, we calculate Spearman's \(\rho\) between rebuttal sentence indices and their aligned review sentence indices. Rebuttals responding to each point in order would achieve \(\rho=1.0\); this case is rare. Many examples with positive \(\rho<1.0\) indicate that authors do respond to points approximately in order, but a simple mapping based on order alone would not capture the correct alignment. Thus, while linear inductive bias may be beneficial to alignment models, the task of determining rebuttal sentences' contexts is not trivial.

[FIGURE:S4.F2][ENDFIGUR

\begin{table}
\begin{tabular}{l l r c} \hline \hline  & \multicolumn{1}{c}{Context type} & \multicolumn{1}{c}{Rebuttal sents.} \\  & & \multicolumn{1}{c}{(Num.)} & \multicolumn{1}{c}{(\%)} \\ \hline \multirow{3}{*}{
\begin{tabular}{r} **Covariant** \\ **Category** \\ \end{tabular} } & Multiple contiguous & 4696 & 42.29\% \\  & Single sentence & 4313 & 38.85\% \\  & Mult. non-contiguous & 407 & 3.67\% \\ \cline{3-4}  & Global context & 816 & 7.35\% \\ \cline{3-4}  & Context in rebuttal & 647 & 5.83\% \\ \cline{3-4}  & No context & 152 & 1.37\% \\ \cline{3-4}  & Context error & 61 & 0.55\% \\ \cline{3-4}  & Cannot be determined & 11 & 0.10\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Different types of rebuttal sentence contexts. Top: Over 84% of sentences are linked to a subset of sentences in the review. Bottom: sentences not linked to any particular subset of review sentences.

Figure 2: Spearman’s \(\rho\) between rebuttal sentence indices and aligned review sentence indices. The dashed line indicates the median \(\rho\) value, which falls at \(0.794\).

 



### Author interpretations of criticism

In our taxonomy, each argumentative rebuttalaction corresponds to a particular review-action, which we refer to as its _canonical_review-action (listed in 'Reply to' column of Table 3). For example, _answers_ are generally responses to _requests_, while _conceding criticism_ is usually a response to an _evaluative_ statement. Annotations revealed that authors often interpreted review sentences as if they embodied reviews besides the canonical one, in a way that furthered the author's argumentative goal. For example, authors often responded to _evaluative_ statements as if they were _requests_, perhaps in order to appease a reviewer, although no action was explicitly requested. Figure 3 shows the distribution of contexts for three different rebuttal-actions.

### Relating discourse features to rating

Figure 4 shows one possible analysis taking into account the rating of the review. We show the distribution of fine-review-action labels of _requests_ with review ratings. It appears that high-scoring manuscripts are rarely asked to add experiments, and are polished enough to not elicit requests to fix typos. Interestingly, low-scoring manuscripts have the second-lowest occurrence of typo requests, which could be due to the preponderance of other requests, but this bears further examination.

## 5 Application: Agreeability

Gao et al. (2019) showed that reviewers do not appear to act upon the rebuttals responding their reviews. It is possible that this is due to paucity of time on the reviewers' part. It is also common practice for area chairs to use review variance across a manuscript's reviews as a practical heuristic to decide which manuscripts need their attention. We propose that discourse information such as that described by DISAPERE can be used to provide heuristics that are data-driven, yet interpretable, and leverage information from the content of reviews rather than just numerical scores, resulting in better decision making.

One such measure is _agreeability_, which we define as the ratio of concur sentences to argumentative sentences in a rebuttal, i.e.: \(aggregility=\frac{n_{comprev}}{n_{conv}+n_{dispute}}\). We argue that low agreeability can indicate problematic reviews even in cases where the variance in scores does not reveal an issue, as illustrated in Figure 5. Agreeability is only weakly correlated with rating, with Pearson's \(r=0.347\). In Figure 5, 18% (28/159) of manuscripts would not meet the bar for high variance scores (top quartile), although their low agreeability (bottom quartile) indicates that they may merit closer attention from area chiars3.

Footnote 3: Two such examples included in DISAPERE: [https://openreview.net/forum?id=rlle?4a4twH](https://openreview.net/forum?id=rlle?4a4twH)

and [https://openreview.net/forum?id=HyMRUiC9YX](https://openreview.net/forum?id=HyMRUiC9YX).

Figure 4: Distribution of review-action labels, separated by rating

Figure 3: Distribution over review-action for the context sentences of three rebuttal-actions. The canonical review-action is marked by cross hatching. Note that authors sometimes interpret requests as criticisms (“Concede criticism”); often respond to evaluative sentences as if they are questions (“Answer”), and sometimes treat criticisms in the form of evaluative sentences as requests which they then carry out. (“Task has been done”)



## 6 Baselines

Two types of machine learning tasks can be defined in DISAPERE. First, a sentence-level classification task for each of the four review labels and the two levels of rebuttal labels. Second, an alignment task in which, given a rebuttal sentence, the set of review sentences that form its context are to be predicted.

The models described below are not intended to introduce innovations in discourse modeling, rather, we intend to show the off-the-shelf performance of state-of-the-art models, and indicate through error analysis the phenomena that are yet to be captured.

### Sentence classification

For the six classification tasks, we use bert-base Devlin et al. (2019) to produce sentence embeddings for each sentence, then classify the representation of the [CLS] token using a feedforward network.

We report macro-averaged F1 scores, shown in Table 7. In general, F1 is lower for tasks with larger label spaces. While the performance is reasonable in most cases, there is still room for improvement. While aspect achieves a particularly low F1 score, its \(\kappa\) is within the bounds of moderate agreement; thus, this must be accounted for by the inherent difficulty of the task rather than a deficit in data quality.

As one might expect, errors in the classification results largely mirror disagreements in the annotations, which in turn reflect particularly ambiguous utterances. One example is the occurrence of rhetorical questions, such as (1) in Table 8, incorrectly labeled as _request_ instead of _evaluative_. In fact, for sentences such as (1), additional context would disambiguate its type: the reviewer answers the question in the next sentence, and hence both sentences were labeled _evaluative_. Similarly, (2) was labeled _fact_, but since it is an integral part of a reviewer's argument against the soundness of the paper, should have been labeled _evaluative_. Certain reviewers also use conventions that do not fit the general schema we observed when developing DISAPERE. For example, (3), an opinionated heading, could be considered both _structuring_ and _evaluative_. Finally, certain lexical cues a model may pick up on can be quite subtle. For example, though they share a prefix, sentences (4) and (5) are clearly _evaluative_ and _request_ respectively.

### Rebuttal context alignment

We model rebuttal context alignment as a ranking task. Ideally, a model should rank all relevant review sentences higher than non-relevant review sentences. As a baseline, we use an information retrieval (IR) model based on BM25 that, given a rebuttal sentence ranks all the corresponding review sentences. We also report results from a neural sentence alignment model based on a two-tower Siamese-BERT (S-BERT) model Reimers and Gurevych (2019). We add a NO_MATCH sentence to the review, to which rebuttal sentences without context sets in the review are aligned. Then, each review and rebuttal sentence is encoded independently using a S-BERT encoder and the similarity between two sentences is computed using cosine

Figure 5: Mean agreeability for a manuscript’s reviews v/s reviewer variance. Manuscripts above the dotted line are in the top quartile of rating variance, and are more likely to be reviewed by area chairs. Manuscripts to the left of the dashed line are in the bottom quartile of mean agreeability, in which authors take issue with the premises of reviewers’ comments. The color of the dots indicates the mean of the reviewers’ ratings.



similarity. We initialize with a model4 pre-trained on various sentence-pair datasets. Alignment is evaluated using mean reciprocal rank (MRR) and Mean Average Precision (MAP).

Footnote 4: We initialize from a sentence-transformers/all-MiniLM-L6-v2 model

Surprisingly, the BM25 model outperforms a neural model Thakur et al. (2021). While this shows that lexical information is a useful signal, both models have significant scope for improvement, and lexical overlap is clearly not sufficient for this task. Importantly, neither of these models account for the context of the rebuttal sentence, and predict each sentence's context independently. Incorporating this information is likely to lead to performance gains; however, we leave this investigation to future work.

## 7 Conclusion

As the burden of academic peer reviewing grows, it is important for program chairs and editors to act upon data-driven insights rather than heuristics, to make the best possible use of participants' scarce time. Models trained on data like DISAPERE will allow decision makers to glean deep insights on the interactions occurring during peer review.

Almost all publicly available peer review data is from the domain of artificial intelligence, limiting the scope of DISAPERE and any similar project. While this means that models trained on DISAPERE won't necessarily generalize to all new domains, we hope that with the detailed annotation guidelines and seamless data collection using the software provided with this paper support, users can build on our work, and ensure that their insights are robust to differences over time and across fields.

## 8 Ethics

The outcomes of peer review can have outsize effects on the careers of participating scholars. As machine learning models are known to amplify biases, we strongly recommend against using the outputs of any machine learning system to make decisions about individual cases. A dataset like DISAPERE is best used to survey participants' behavior. Any interventions based on this information should be subjected to studies in order to ensure that they do not introduce or exacerbate bias.

## Acknowledgments

This material is based upon work supported in part by the National Science Foundation under Grant Numbers IIS-1763618, IIS-1922090, and IIS-1955567, in part by the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C-0106 under Subaward No. 89341790 from the University of Southern California, in part by the Office of Naval Research (ONR) via Contract No. N660011924032 under Subabward No. 123875727 from the University of Southern California, in part by IBM Research AI through the AI Horizons Network, in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction, and in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.

## References

* Ajjour et al. (2017) Yamen Ajjour, Wei-Fan Chen, Johannes Kiesel, Henning Wachsmuth, and Benno Stein. 2017. Unit segmentation of argumentative texts. In _Proceedings of the 4th Workshop on Argument Mining_, pages 118-

\begin{table}
\begin{tabular}{l l c} \hline \hline  & S-BERT & BM25 \\ \hline MAP & 0.4409 & 0.5174 \\ MRR & 0.5022 & 0.5980 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Rebuttal context alignment results. The results of both models indicate significant scope for improvement.

 

128, Copenhagen, Denmark. Association for Computational Linguistics.
* Bao et al. (2021) Jianzhu Bao, Bin Liang, Jingyi Sun, Yice Zhang, Min Yang, and Ruifeng Xu. 2021. Argument pair extraction with mutual guidance and inter-sentence relation graph. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3923-3934, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Chakraborty et al. (2020) Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_.
* Cheng et al. (2020) Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7000-7011, Online. Association for Computational Linguistics.
* Cheng et al. (2021) Liying Cheng, Tianyu Wu, Lidong Bing, and Luo Si. 2021. Argument pair extraction via attention-guided multi-layer multi-cross encoding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6341-6353, Online. Association for Computational Linguistics.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fromm et al. (2020) Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Rucoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2020. Argument mining driven analysis of peer-reviews. _arXiv preprint arXiv:2012.07743_.
* Gao et al. (2019) Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major NLP conference. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1274-1290, Minneapolis, Minnesota. Association for Computational Linguistics.
* Habernal and Gruevych (2017) Ivan Habernal and Iryna Gruevych. 2017. Argumentation mining in user-generated web discourse. _Computational Linguistics_, 43(1):125-179.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Hua et al. (2019) Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2131-2137, Minneapolis, Minnesota. Association for Computational Linguistics.
* Miller et al. (2019) Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A streamlined method for sourcing discourse-level argumentation annotations from the crowd. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1790-1796, Minneapolis, Minnesota. Association for Computational Linguistics.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108, Online. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Soergel et al. (2013) David Soergel, Adam Saunders, and Andrew McCallum. 2013. Open scholarship and peer review: a time for experimentation.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.
* Trautmann et al. (2020) Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schutze, and Iryna Gurevych. 2020. Fine-grained argument unit recognition and classification. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 9048-9056.
* Yuan et al. (2021) Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021. Can we automate scientific reviewing?
* Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using 

corpus-level constraints. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguistics.

## Appendix A Rationale for taxonomy construction

Our label sets leverage ideas from and comonalities between existing work in this domain, including AMPERE Hua et al. (2019), AMSR Fromm et al. (2020) ASAP-Review Yuan et al. (2021), and Gao et al. (2019):

* ASAP-Review's polarity labels approximately correspond to _arg-pos_ and _arg-neg_ labels in AMSR
* AMSR and AMPERE each label non-argumentative sentences in a similar manner
* _aspect_ labels from ASAP-Review apply only to certain types of sentences; namely _request_ and _evaluative_ sentences from AMPERE's taxonomy.
* _summary_ is an exception among ASAP-Review's _aspect_s, behaving similarly to AMPERE's _quote_. We thus include both of these under a _structuring_ category.
* Further, in order to gauge the extent to which authors acquiesced to reviewers' requests, we introduce a fine-grained categorization of the types of requests.
* Gao et al. (2019) enumerates some features of rebuttals, including expressing gratitude, promising revisions, and disagreeing with criticisms. We formalize these observations into our rebuttal label taxonomy.

## Appendix B Annotation tool

Two modes of annotation are possible. First, annotators can apply labels on a sentence-by-sentence basis. Multiple labeling schemas can be annotated simulatenously, with the option of adding constraints so that certain values govern possible values for other properties. This annotation mode is shown in Figure 6.

The second annotation mode can build on the output of the first annotation mode. Here, sentences of a focus text (the rebuttal) are presented in sequence, and annotators are permitted to select one or more of the sentences in the reference text (the review) which form the context of the sentence of the focus text. Further, a label can be applied to the alignment. This annotation mode is shown in Figure 7 and Figure 8 



## Appendix C Annotated review-rebuttal pair

Figure 9 shows a truncated version of a review-rebuttal pair from the train set of DISAPERE.

## Appendix D Context overlap analysis

As a proxy for agreement of rebuttal spans, we show the types of overlap between spans on rebuttal sentences from 81 examples annotated by two annotators in Table 10.

## Appendix E Additional Agreement Analysis

While some of the IAA scores on annotation are low, we note that the labels used in this task attempt to characterize relatively complex relationships in text. To give more insight into such disagreements, Figure 10 provides a confusion matrix regarding the rebuttal-action labels. Recognizing that there are often situations in which users of a dataset will hope to reduce a label set, we provide some guidance as to which such merges may be acceptable and which are not.

Many disagreements come from three labels which might be said to exist upon a continuum - answer, mitigate criticism and reject criticism. We suggest that in the situation of needing to minimize IAA disagreement, one might consider first merging _mitigate criticism_ into _reject criticism_. The kind of disagreements seen between the two are understandable but nuanced: the difference between saying that the reviewer has a point (but that they disagree on the relevance of that point) and disagreeing with the point itself. Out-of-context rebuttal sentences illustrating this are provided below as examples of this kind of ambiguous situation:

\begin{table}
\begin{tabular}{l l l} \hline \hline Type of & Num. rebuttal & \% rebuttal \\ context overlap & sentences & sentences \\ \hline Exact match & 914 & 53.11\% \\ Partial match & 492 & 28.59\% \\ Agree none & 122 & 7.09\% \\ Disagree none & 100 & 5.81\% \\ No overlap & 93 & 5.40\% \\ \hline \hline \end{tabular}
\end{table}
Table 10: Types of context overlap. Full agreement is achieved in the top rows (exact match and ‘Agree none’, where both annotators agree that there is no appropriate subset of review sentences forming the context. in ‘Disagree none’, one annotator marks a subset of review sentences, while the other does not.

Figure 6: Review annotation interface. Annotators select label values from dropdown menus for each review sentence in turn. [1] Title of the manuscript whose review is being annotated [2] Reviewer identifier [3] Annotator identifier (removed for anonymity) [4] Link to original forum, in case it is needed for context [5] Individual review sentence [6] Option to report sentence splitting error (sentence splitting generally suffered from false positives) [7] Dropdown for review-action [8] Follow-up dropdownfor fine-review-action populated based on value in (7) [9] Button to add second review-action if necessary (this was seldom used)

* [10] M. C. Dickson, A. K. Sridharan, and R. J. Schuster, "A review of the design of a new class of multi-agent systems for multi-agent systems," in _Proceedings of the IEEE Conference on Decision and Control (CDC)_, pp. 2357-2368, 2014.

[MISSING_PAGE_POST]

 

* _We note that such rules are indeed limited to some extent, but they still capture a rather expressive fragment of answer set programs with restricted forms of external computations._
* _The use of_ \(C^{val}_{p}\) _for hyperparameter tuning was incidental and not a central point of our paper._
* clipped actions._

Furthermore, we note that (as illustrated in the confusion matrix) a wide range of disagreements are hard to distinguish from "answer" labels, as authors often attempt to frame disagreements as simple answers to questions.

Figure 8: Rebuttal annotation interface (continued from Figure 7). Annotators examine each rebuttal sentence in turn, selecting sentences as context specifying rebuttal-action.[1] Full review text, continued from (7) in Figure 7.[2] Dropdown to select context type, in case context cannot be defined as a subset of review sentences.[3] Dropdown to select rebuttal-action (keyboard navigation possible)[4] Buttons to select rebuttal-action (in case mouse navigation is preferred)[5] Option to report egregious sentence splitting errors.

 

* [ "metadata": {  "Forum_id": ""ByGWhJBtDB",  "review_id": "BJqmhEfTcH",  "rebuttal_id": "rye3za27or",  "title": "Hyperparameter Tuning and Implicit Regularization in Minibatch SGD",  "reviewer": "AnonReviewer3", "rating": 3, "conference": "ICLR2020",  "permalink": "[https://openreview.net/forum?id=ryGWhJBtDB&noteId=rye3za27or](https://openreview.net/forum?id=ryGWhJBtDB&noteId=rye3za27or)",  "annotator": "annol0"  ),  "review_sentences": [  {  "review_id": "BQmhEfTcH",  "sentence_index": 0,  "text": "This paper is an empirical contribution regarding SGD arguing that it presents two different behaviors which the authors name a noise dominated regimen, and a curvature dominated regime.",  "suffix": "",  "review_action": "arg_structuring", "fine_review_action": "arg-  structuring_summary",  "aspect": "none", "polarity": "none"  },  ...  {  "review_id": "BQmhEfTcH",  "sentence_index": 4,  "text": "I find the observations interesting, but the contribution is  empirical and not entirely new. It would be nice if there were some  theoretical results to back up the observations.",  "suffix": "",  "review_action": "arg_evaluative", "fine_review_action": "none",  "aspect": "asp_originality", "polarity": "pol_negative"  },  ],  "rebuttal_sentences": [  {  "review_id": "BQmphEfTcH", "rebuttal_id": "rye3za27or",  "sentence_index": 0,  "text": "We thank the reviewer for their comments.",  "suffix": "\n\n",  "rebuttal_stance": "nonary", "rebuttal_action": "rebuttal_social",  "alignment": [ "context_global", null] },  {  "review_id": "BQmhEfTcH", "rebuttal_id": "rye3za27or",  "sentence_index": 1,  "text": "Although our primary contributions are empirical, we also provided  a detailed theoretical discussion in section 2, where we give a clear  and simple account of why the two regimes arise.",  "suffix": "",  "rebuttal_stance": "dispute", "rebuttal_action": "rebuttal_regject-criticism  ",  "alignment": ["context_sentences", [4]]  },  ...  ]

Figure 9: A (truncated) example from the training set of DISAPERE.

 

* [10] M. C. Dickson, J. R. Friedman, and A. K. S. Guth, "Theoretical Statistics of the New York: The Case of Statistical Issues," _Annual Review of Economics_**26**(4), no. 3, pp. 57-89, 1996.
* [11] P. W. Anderson, "Statistical Theory of Random Fields," _Journal of Statistical Physics_**10**(1), no. 1, p. 105, 1970.
* [12] T. B. Cover, "Random Fields," in _Proceedings of the National Academy of Sciences_**10**(1), no. 1, p. 105, 1970.

[MISSING_PAGE_POST]