SlowGuess/NAACL_2022_OCR huggingface dataset:
Neha Nayak Kennard Tim O'Gorman Rajarshi Das Akshay Sharma Chhandak Bagchi Matthew Clinton Pranay Kumar Yelugam Hamed Zamani Andrew McCallum University of Massachusetts Amherst
{kennard, togorman, rajarshi, akshaysharma, cbagchi, mfclinton, pyelugam, zamani, mccallum}@cs.umass.edu
At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined.
We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors' stance towards review arguments. Further, we annotate every review and rebuttal sentence.
We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
Peer review performs the essential role of quality control in the dissemination of scientific knowledge. The recent rapid increase in academic output places an immense burden on decision makers such as area chairs and editors, as their decisions must take into account not only extensive manuscripts, but enormous additional amounts of technical text including reviews, rebuttals, and other discussions.
One long term goal of research in peer review is to support decision makers in managing their workload by providing tools to help them efficiently absorb the discussions they must read. While machine learning should not be used to produce condensed accounts of the peer review text due to the risk of amplifying biases (Zhao et al., 2017), ML tools could nevertheless help manage information overload by identifying patterns in the data, such as argumentative strategies, goals, and intentions.
Any such research requires an extensive labeled dataset. While the OpenReview platform (Soergel et al., 2013) has made it easy to obtain unlabeled public peer review text, labeling this data for supervised NLP requires highly qualified annotators. Correct interpretation of the discourse structure of the text requires an understanding of the technical content, precluding the use of standard crowdsourcing techniques. Prior work on discourse in peer review has focused this qualified labor force on labeling arguments extracted from the text, which enables the complete annotation of more examples, at the expense of research on non-argumentative behaviors in peer review. While there has been extensive research and deep analysis of different aspects of peer review, the taxonomies used to describe review argumentation are disparate and not directly compatible. Finally, there has been limited research into understanding the discourse relations between rebuttals and reviews (Cheng et al., 2020; Bao et al., 2021), and none so far into the discourse structure of rebuttals.
This paper presents DISAPERE (DIScourse Structure in Academic PEer REview), a dataset focusing on the interaction between reviewer and author<sup>1</sup>. We give reviews and rebuttals equal importance, and emphasize the relations between them. To enable the study of behaviors beyond the core arguments, we also annotate every sentence of both the review and rebuttal, and provide fine-grained labels for non-argumentative types. We annotate at the sentence level not only for completeness but also to avoid the propagation of errors from argument detection. We annotate four properties (REVIEW-ACTION, FINE-REVIEW-ACTION, ASPECT, POLARITY) of each review sentence, where the set of properties and their values were developed by synthesizing taxonomies from
<sup>1</sup>The dataset, along with its accompanying code and documentation, is available at http://www.github.com/nnkennard/DISAPERE/.
Human Language Technologies, pages 1234 - 1249
Figure 1: A depiction of our annotation scheme on a minimal, fictional review-rebuttal pair. A: REVIEW-ACTION, including Structuring, Request, Evaluation; B: FINE-REVIEW-ACTION, fine-grained categorization of Structuring and Request sentences; C: ASPECT, indicating the qualities of the manuscript being commented upon D: POLARITY indicating whether these comments are positive or negative in nature. E: Each sentence in the rebuttal is mapped to zero or more sentences in the review, which constitute its context. This is a many-to-many relation. F: The sentences in the rebuttal are labeled with domain-specific discourse acts (REBITTAL-ACTION); each discourse act may be categorized according to whether it concurs with  $(\checkmark)$  or disputes  $(\times)$  the premise of the context it is responding to.
prior work. We also annotate each sentence of a rebuttal with a fine-grained label indicating the author's intentions and commitment, and a link to the set of review sentences that form its context. Figure 1 shows the DISAPERE annotation scheme on a minimal, fictional example review-rebuttal pair.
DISAPERE is intended as a comprehensive and high-quality test collection, along with training data to fine-tune models. Our annotations are carried out by graduate students in computer science who have undergone training and calibration, amounting to over 850 person-hours of annotation work. Much of the test data is double-annotated, and we report inter-annotator agreement on all aspects of the annotation. We describe the performance of state of the art models on the tasks of predicting labels and contexts, showing that interesting ambiguities in the data provide the NLP community with research challenges. We also show an example that demonstrates how decision makers could use models like these to understand trends and inform policies for future conferences (§ 5).
The contributions of this paper are as follows: (1) a new labeled training dataset of 506 reviewrebuttal pairs (over 20k sentences) of peer review discussion text in English, where review sentences are annotated with four properties, and rebuttal sentences are annotated with context and labels from a novel scheme to describe discourse structure; (2) a taxonomy of discourse labels synthesizing prior work on discourse in peer review and extending it to add useful subcategories; (3) a summary of the performance of baseline models on the dataset  $(\S 6)$ ; (4) examples of analyses on the dataset that
could benefit peer review decision makers (§§ 4 and 5), and (5) extensive annotation guidelines and software to support future labeling efforts.
The design of this dataset draws upon extensive, but disparate prior work on this topic. Many works, some addressed below, have taken advantage of the availability of review text hosted on OpenReview.
Argument-level review labeling Prior work has developed label sets that address different phenomena. Hua et al. (2019) introduced the study of discourse structure in peer review by annotating argumentative propositions in the AMPERE dataset with a set of labels tailored to the peer review domain (EVALUATION, REQUEST, FACT, REFERENCE, and QUOTE). Similarly, Fromm et al. (2020)'s AMSR dataset frames the problem as an argumentation process, in which the stance of each argument towards the paper's acceptance or rejection is of paramount importance. Both view peer review as argumentation, using argument mining techniques to highlight spans of interest.
While its goal is not to examine discourse structure per se, Yuan et al. (2021) uses polarity labels to indicate each argument's support or attack of the authors' bid for acceptance. Besides polarity, these examples follow Chakraborty et al. (2020) by annotating each argument with the aspect of the paper it comments on. In contrast to Yuan et al. (2021), we do not attempt or recommend generat
2Aspects are based on the ACL 2018 rubric.
ing peer review text, instead focusing on analyzing human-generated text in peer review.
Review-rebuttal interactions We also expand on work by Cheng et al. (2020), who first annotated discourse relations between sentences in reviews and rebuttals. While Cheng et al. (2020, 2021) present new deep learning architectures, in this paper we focus on the creation and comprehensive annotation of a new dataset, illustrated with results from some less specialized baseline models.
Other research into rebuttals includes Gao et al. (2019). Besides their main finding that reviewers rarely change their rating in response to rebuttals, they find that more specific, convincing and explicit responses are more likely to elicit a score change. Observations from this paper are formalized into rebuttal action labels in DISAPERE.
Comparison of datasets In DISAPERE we attempted to unify these schemas to form a single hierarchical schema for review discourse structure. We then expanded this hierarchical schema to introduce fine-grained classes for implicit and explicit requests made by the reviewers. The details of the correspondence between DISAPERE labels and those from prior work are summarized in Appendix A. In contrast to prior work, DISAPERE labels discourse phenomena at the sentence level rather than the argument level. This enables more thorough coverage of the text while avoiding the propagation of errors from machine learning models earlier in the annotation pipeline. While using manually defined discourse units (above or below the sentence level) may more precisely capture some discourse information, a separate pass of discourse segmentation can hinder the use of discourse datasets, as achieving consistent and replicable annotation of argument units is known to be highly challenging (Trautmann et al., 2020), and also because few works actually tackle unit segmentation (Ajjour et al., 2017).
Each example in DISAPERE consists of a pair of texts: a review and a rebuttal. Labels for reviews and rebuttal sentences are described below. Review sentence labels are summarized in Table 2, and rebuttal sentence labels in Table 3.
Dataset	AMPERE	AMSRR	ASAP-Review	APE	DISAPERE
# examples	400	77	1k	4.7k	506
# labels	10k	1.4k	5.7k	130k	46k
Review	Arg. stms.	✓	✓	✓	✓
Arg. types	✓	✓
Polarity	✓	✓	✓
Aspect	✓	✓
Non-arg.	✓
All sents.	✓
Rebuttal	Included?	✓	✓
Arg. stms.	✓	✓
Context	✓	✓
Arg. types	✓
Non-arg.	✓
All sents.	✓
Table 1: Comparison between our dataset and prior work: AMPERE (Hua et al., 2019), AMSR (Fromm et al., 2020), ASAP-Review (Yuan et al., 2021), APE (Cheng et al., 2020). Arg.stmts.: Are argumentative statements highlighted?; Arg. types: Are subtypes of argumentative statements labeled?; Non-arg: Are non-argumentative statements labeled?; All sents.: Are labels provided for all sentences?; Context: Are rebuttal texts' contexts in the review provided? DisAPERE is the only work to annotate every sentence in the review and rebuttal, and the only work that applies discourse labels to the author's actions in the rebuttal.
3.1 Review sentence labels
3.1.1 Review actions
REVIEW-ACTION annotations characterize a sentence's intended function in the review. Annotators label each sentence with one of six coarse-grained sentence types including evaluative and fact sentences, request sentences (including questions, which are requests for information), as well as non-argument types: social, and structuring for organization of the text.
3.1.2 Fine-grained review actions
We also extend two of these review actions with subtypes: structuring sentences include headers, quotations, or summarization sentences, and request sentences are subdivided by the nature of the request, distinguishing between clarification of factual information, requests for new experiments,
Category	Label	Description	Percentage
REVIEW-ACTION	Evaluative	A subjective judgement of an aspect of the paper	32.83%
Structuring	Text used to organize an argument	27.70%
Request	A request for information or change in regards to the paper	19.82%
Fact	An objective truth, typically used to support a claim	8.55%
Social	Non-substantive text typically governed by social conventions	1.41%
Other	All other sentences	9.71%
ASPECT	Substance	Are there substantial experiments and/or detailed analyses?	17.09%
Clarity	Is the paper clear, well-written and well-structured?	11.08%
Soundness/Correctness	Is the approach sound? Are the claims supported?	9.58%
Originality	Are there new topics, technique, methodology, or insights?	3.85%
Motivation/Impact	Does the paper address an important problem?	3.69%
Meaningful Comparison	Are the comparisons to prior work sufficient and fair?	3.15%
Replicability	Is it easy to reproduce and verify the correctness of the results?	2.86%
POLARITY	Negative	Negatively describes an aspect of the paper (reason to reject)	29.43%
Positive	Positively describes an aspect of the paper (reason to accept)	11.16%
FINE-REVIEW-ACTION	Struct.	Summary	Reviewer's summary of the manuscript	18.17%
Heading	Text used to organize sections of the review	8.54%
Quote	A quote from the manuscript text	1.00%
Request	Explanation	Request to explain scientific choices (question)	5.50%
Experiment	Request for additional experiments or results	4.78%
Edit	Request to edit the text in the manuscript	4.14%
Clarification	Request to clarify the meaning of some text (question)	2.80%
Typo	Request to fix a typo in the manuscript	1.98%
Table 2: A list of the review sentence labels, their descriptions, and the percentage of review sentences they apply to. Labels from all categories besides REVIEW-ACTION are optional, and thus may not add up to  $100\%$ .
requests for an explanation (e.g. of motivations or claims), requests for edits, and identification of minor typos.
3.1.3 Aspect and polarity
ASPECT annotations follow the ACL review form (Chakraborty et al., 2020; Yuan et al., 2021). These distinguish clarity, originality, soundness/correctness, replicability, substance, impact/motivation, and meaningful comparison. Following Yuan et al. (2021), arguments with an ASPECT are also annotated for POLARITY. We label positive and negative polarities. ASPECT and POLARITY are applied to sentences whose REVIEW-ACTION value is evaluative or request.
3.2 Rebuttal sentence labels
We annotate two properties of each rebuttal sentence: a REBITTAL-ACTION label characterizing its intent, and its CONTEXT in the review in the form of a subset of review sentences.
3.2.1 Rebuttal actions
The 14 rebuttal actions (Table 3) are divided into three REBUTTAL-STANCE categories (concur, dis
pute, non-arg) based on the author's stance towards the reviewer's comments.
(1) concur: The author concurs with the premise of the context. This includes answering a question or discussing a requested change that has been made to the manuscript, conceding a criticism in an evaluative sentence. (2) dispute: The author disputes the premise of the context. The rebuttal sentence may reject a criticism or request, disagree with an underlying fact or assertion, or mitigate criticism (accepting a criticism while, e.g., arguing it to be offset by other properties). (3) non-arg: Encompasses rebuttal actions including social actions (such as thanking reviewers), and structuring labels, for sentences that organize the review.
Responses to requests are further annotated: if the author concurs, we record whether the task has been completed by the time of the rebuttal, or promised by the camera ready deadline; if the author disputes, we record whether the task was deemed to be out of scope for the manuscript.
3.2.2 Rebuttal context
We refer to the set of sentences which a rebuttal sentence is responding to as the context of that
Category	Label	Description	Reply to	Percentage
Argumentative	Concur	Answer	Answer a question	Request	32.76%
Task has been done	Claim that a requested task has been completed	Request	8.58%
Concede criticism	Concede the validity of a negative eval. statement	Evaluative	2.70%
Task will be done	Promise a change by camera ready deadline	Request	2.01%
Accept for future work	Express approval for a suggestion, but for future work	Request	1.30%
Accept praise	Thank reviewer for positive statements	Evaluative	0.35%
Dispute	Reject criticism	Reject the validity of a negative eval. statement	Evaluative	10.37%
Mitigate criticism	Mitigate the importance of a negative eval. statement	Evaluative	2.43%
Reject request	Reject a request from a reviewer	Request	1.16%
Refute question	Reject the validity of a question	Request	0.95%
Contradict assertion	Contradict a statement presented as a fact	Fact	0.86%
Non-arg	Structuring	Text used to organize sections of the review	-	17.82%
Summary	Summary of the rebuttal text	-	7.94%
Social	Non-substantive social text	-	6.71%
Followup question	Clarification question addressed to the reviewer	-	0.32%
Other	All other sentences	-	3.75%
sentence, with special labels for when referring to the entire review (global context) or the empty set (no context). By not mandating a fixed discourse chunking, these annotations may handle situations when some rebuttal sentences respond to large sections of text, and other rebuttal sentences respond to specific sentences within those sections.
3.3 Data Source and Annotation
DISAPERE uses English text from scientific discussions on OpenReview (Soergel et al., 2013), which makes peer review reports available for research purposes. We draw review-rebuttal pairs from the International Conference on Learning Representations (ICLR) in 2019 and 2020, resulting in text within the domain of machine learning research. Review-rebuttal pairs are split into train, development and test sets in a 3:1:2 ratio such that all texts associated with any manuscript occur in the same subset. Overall statistics for the dataset are summarized in Table 4.
Authors are able to respond to each ICLR review by adding a comment. Although rebuttals are not formally named, we consider direct replies by the author to the initial review comment to constitute a rebuttal. While multi-turn interactions are possible, we focus on reviews and initial responses, and leave study of extended discussion for future work. The text is separated into sentences using the spaCy (Honnibal and Montani, 2017) sentence separator.
Annotation was accomplished with a custom
Table 3: A list of the rebuttal sentence labels, their descriptions, and the percentage of rebuttal sentences they apply to. The "Reply to" column shows the REVIEW-ACTION types that a particular rebuttal type would canonically reply to. Each rebuttal sentence has exactly one REBITTAL-ACTION label, so these percentages add up to  $100\%$ .
Train	Dev	Test
Num. review-rebuttal pairs	251	88	167
Num. manuscripts	94	37	57
Num. adjudicated pairs	0	0	65
Num. review sentences	5216	1484	3246
Num. rebuttal sentences	5805	2015	3283
Num. review tokens	112k	33k	70k
Num. rebuttal tokens	131k	44k	75k
Table 4: Statistics for the dataset. Where possible, multiple reviews for the same manuscript were annotated. All reviews for any particular manuscript fall within the same train/dev/test split. Adjudicated pairs are those that were annotated by multiple annotators, and had disagreements resolved by an experienced annotator. All test set pairs are double-annotated. While the original sentence boundaries were maintained, tokenization within sentences was carried out using Stanza(Qi et al., 2020).
annotation tool designed for this task, which is available as part of the code release accompanying DISAPERE. The tool is described in detail in Appendix B. Annotators annotate each sentence of a review, then examine the rebuttal sentences in order, selecting sets of review sentences to form their context. While this linking between sentences does not explicitly align multi-sentence chunks as in pipelined approaches to discourse alignment (Cheng et al., 2020), we note that since multiple sentences may be aligned to the same set of sen
tences in the review, some discourse structure is nevertheless latently implied.
3.4 Agreement
We report Cohen's  $\kappa$  (Cohen, 1960) on the IAA of labeling both review and rebuttals, treating each sentence as a labeling unit (Table 5). The annotators for each example are selected randomly from the pool of 10 annotators. Cohen's  $\kappa$  is calculated for sentences annotated at least twice. Where more than two annotations were produced, we calculate  $\kappa$  between all pairs and normalize by the number of possible pairs. The results show between moderate and substantial chance-corrected agreement between annotators, for both REBITTAL-ACTION and REBITTAL-STANCE labels (Appendix D provides details about agreement on context sentences). While these IAA scores do illustrate the noise of the task, note that this is not highly unusual for discourse labeling tasks - e.g. Habernal and Gurevych (2017) and Miller et al. (2019) both report  $\alpha_{u}$  between 0.4 and 0.5.
Label	Cohen's κ
REVIEW-ACTION	0.605
FINE-REVIEW-ACTION	0.583
ASPECT	0.447
POLARITY	0.561
REBUTTAL-STANCE	0.513
REBUTTAL-ACTION	0.479
4.1 Context types
We separate the different types of rebuttal contexts in terms of the number and relative position of selected review sentences in Table 6, along with the four cases in which the context cannot be described as a subset of review sentences. Notably,  $84.81\%$  of sentences are linked to some review context. A small number of sentences refer to other sentences within the rebuttal, rather than any review context, posing a challenge for future work.
Table 5: IAA for review labels (top) and rebuttals (bottom), scored on double annotation. IAA is reported on 65 double-annotated examples, all of which fall in the test set of DISAPERE.
Context type	Rebuttal sents.
(Num.)	(%)
Sents.	Multiple contiguous	4696	42.29%
Single sentence	4313	38.85%
Mult. non-contiguous	407	3.67%
No sents.	Global context	816	7.35%
Context in rebuttal	647	5.83%
No context	152	1.37%
Context error	61	0.55%
Cannot be determined	11	0.10%
Table 6: Different types of rebuttal sentence contexts. Top: Over  $84\%$  of sentences are linked to a subset of sentences in the review. Bottom: sentences not linked to any particular subset of review sentences.
4.2 Alignment
One might reasonably hypothesize that the task of alignment between rebuttal and review sentences would be trivial, since authors are likely to respond to each point in the review in order. We can show that this is not the case. In Figure 2, we calculate Spearman's  $\rho$  between rebuttal sentence indices and their aligned review sentence indices. Rebuttals responding to each point in order would achieve  $\rho = 1.0$ ; this case is rare. Many examples with positive  $\rho < 1.0$  indicate that authors do respond to points approximately in order, but a simple mapping based on order alone would not capture the correct alignment. Thus, while linear inductive bias may be beneficial to alignment models, the task of determining rebuttal sentences' contexts is not trivial.
Figure 2: Spearman's  $\rho$  between rebuttal sentence indices and aligned review sentence indices. The dashed line indicates the median  $\rho$  value, which falls at 0.794.
Figure 3: Distribution over REVIEW-ACTION for the context sentences of three REBUTTAL-ACTIONS. The canonical REVIEW-ACTION is marked by cross hatching. Note that authors sometimes interpret requests as criticisms ("Concede criticism"); often respond to evaluative sentences as if they are questions ("Answer"), and sometimes treat criticisms in the form of evaluative sentences as requests which they then carry out. ("Task has been done")
4.3 Author interpretations of criticism