SlowGuess/NAACL_2022_OCR huggingface dataset:
In our taxonomy, each argumentative REBUTTAL-ACTION corresponds to a particular REVIEW-ACTION, which we refer to as its canonical REVIEW-ACTION (listed in 'Reply to' column of Table 3). For example, answers are generally responses to requests, while conceding criticism is usually a response to an evaluative statement. Annotations revealed that authors often interpreted review sentences as if they embodied REVIEW-ACTIONS besides the canonical one, in a way that furthered the author's argumentative goal. For example, authors often responded to evaluative statements as if they were requests, perhaps in order to appease a reviewer, although no action was explicitly requested. Figure 3 shows the distribution of contexts for three different REBUTTAL-ACTIONS.
4.4 Relating discourse features to rating
Figure 4 shows one possible analysis taking into account the rating of the review. We show the distribution of FINE-REVIEW-ACTION labels of requests
Figure 4: Distribution of REVIEW-ACTION labels, separated by rating
with review ratings. It appears that high-scoring manuscripts are rarely asked to add experiments, and are polished enough to not elicit requests to fix typos. Interestingly, low-scoring manuscripts have the second-lowest occurrence of typo requests, which could be due to the preponderance of other requests, but this bears further examination.
5 Application: Agreeability
Gao et al. (2019) showed that reviewers do not appear to act upon the rebuttals responding their reviews. It is possible that this is due to paucity of time on the reviewers' part. It is also common practice for area chairs to use review variance across a manuscript's reviews as a practical heuristic to decide which manuscripts need their attention. We propose that discourse information such as that described by DISAPERE can be used to provide heuristics that are data-driven, yet interpretable, and leverage information from the content of reviews rather than just numerical scores, resulting in better decision making.
One such measure is agreeability, which we define as the ratio of CONCUR sentences to argumentative sentences in a rebuttal, i.e.: agreeability  $= \frac{n_{concur}}{n_{concur} + n_{dispute}}$ . We argue that low agreeability can indicate problematic reviews even in cases where the variance in scores does not reveal an issue, as illustrated in Figure 5. Agreeability is only weakly correlated with rating, with Pearson's  $r = 0.347$ . In Figure 5,  $18\%$  (28/159) of manuscripts would not meet the bar for high variance scores (top quartile), although their low agreeability (bottom quartile) indicates that they may merit closer attention from area chairs<sup>3</sup>.
3Two such examples included in DISAPERE: https://openreview.net/forum?id=r1e74a4twH and https://openreview.net/forum?id=HyMRUiC9YX.
Figure 5: Mean agreeability for a manuscript's reviews v/s reviewer variance. Manuscripts above the dotted line are in the top quartile of rating variance, and are more likely to be reviewed by area chairs. Manuscripts to the left of the dashed line are in the bottom quartile of mean agreeability, in which authors take issue with the premises of reviewers' comments. The color of the dots indicates the mean of the reviewers' ratings.
6Baselines
Two types of machine learning tasks can be defined in DISAPERE. First, a sentence-level classification task for each of the four review labels and the two levels of rebuttal labels. Second, an alignment task in which, given a rebuttal sentence, the set of review sentences that form its context are to be predicted.
The models described below are not intended to introduce innovations in discourse modeling, rather, we intend to show the off-the-shelf performance of state-of-the-art models, and indicate through error analysis the phenomena that are yet to be captured.
6.1 Sentence classification
For the six classification tasks, we use bert-base (Devlin et al., 2019) to produce sentence embeddings for each sentence, then classify the representation of the [CLS] token using a feedforward network.
We report macro-averaged F1 scores, shown in Table 7. In general, F1 is lower for tasks with larger label spaces. While the performance is reasonable in most cases, there is still room for improvement. While ASPECT achieves a particularly low F1 score, its  $\kappa$  is within the bounds of moderate agreement; thus, this must be accounted for by the inherent difficulty of the task rather than a deficit in data quality.
Classification task	Macro F1 (test)	Cohen's κ	Num. labels
REVIEW-ACTION	60.42%	0.605	7
FINE-REVIEW-ACTION	44.83%	0.583	10
ASPECT	38.28%	0.447	9
POLARITY	70.88%	0.561	3
REBUTTAL-STANCE	43.36%	0.513	4
REBUTTAL-ACTION	31.23%	0.479	17
Table 7: Sentence classification results. Top: review labels; Bottom: rebuttal labels.
As one might expect, errors in the classification results largely mirror disagreements in the annotations, which in turn reflect particularly ambiguous utterances. One example is the occurrence of rhetorical questions, such as (1) in Table 8, incorrectly labeled as request instead of evaluative. In fact, for sentences such as (1), additional context would disambiguate its type: the reviewer answers the question in the next sentence, and hence both sentences were labeled evaluative. Similarly, (2) was labeled fact, but since it is an integral part of a reviewer's argument against the soundness of the paper, should have been labeled evaluative. Certain reviewers also use conventions that do not fit the general schema we observed when developing DISAPERE. For example, (3), an opinionated heading, could be considered both structuring and evaluative. Finally, certain lexical cues a model may pick up on can be quite subtle. For example, though they share a prefix, sentences (4) and (5) are clearly evaluative and request respectively.
6.2 Rebuttal context alignment
We model rebuttal context alignment as a ranking task. Ideally, a model should rank all relevant review sentences higher than non-relevant review sentences. As a baseline, we use an information retrieval (IR) model based on BM25 that, given a rebuttal sentence ranks all the corresponding review sentences. We also report results from a neural sentence alignment model based on a two-tower Siamese-BERT (S-BERT) model (Reimers and Gurevych, 2019). We add a NO_MATCH sentence to the review, to which rebuttal sentences without context sets in the review are aligned. Then, each review and rebuttal sentence is encoded independently using a S-BERT encoder and the similarity between two sentences is computed using cosine
Label (Pred.)
1	Can the proposed [...] function represent all function the authors used in the paper? Yes.	E (R)
2	Matrices can have either “horizontal” or “vertical” redundancy (or “other” or neither).	E (F)
3	Solid technical innovation/contribution:	E
4	I am also wondering if the comparison with the baselines is fair.	E
5	I wonder if the authors ever looked at how much [...] determines the performance of the system?	R
similarity. We initialize with a model $^4$  pre-trained on various sentence-pair datasets. Alignment is evaluated using mean reciprocal rank (MRR) and Mean Average Precision (MAP).
Table 8: Example sentences including errors and challenging cases. E, R, F stand for evaluative, request and fact respectively. Letters in parentheses show the incorrect label from the model. Sentence (3) functions both as evaluative and structuring. Sentences (4) and (5) share a prefix but have different REVIEW-ACTIONS.
S-BERT	BM25
MAP	0.4409	0.5174
MRR	0.5022	0.5980
Table 9: Rebuttal context alignment results. The results of both models indicate significant scope for improvement.
Surprisingly, the BM25 model outperforms a neural model (Thakur et al., 2021). While this shows that lexical information is a useful signal, both models have significant scope for improvement, and lexical overlap is clearly not sufficient for this task. Importantly, neither of these models account for the context of the rebuttal sentence, and predict each sentence's context independently. Incorporating this information is likely to lead to performance gains; however, we leave this investigation to future work.
As the burden of academic peer reviewing grows, it is important for program chairs and editors to act upon data-driven insights rather than heuristics, to make the best possible use of participants' scarce time. Models trained on data like DISAPERE will allow decision makers to glean deep insights on the interactions occurring during peer review.
Almost all publicly available peer review data is from the domain of artificial intelligence, limiting the scope of DISAPERE and any similar project. While this means that models trained on DISAPERE won't necessarily generalize to all new domains, we hope that with the detailed annotation guidelines and seamless data collection using the software provided with this paper support, users
can build on our work, and ensure that their insights are robust to differences over time and across fields.
8 Ethics
The outcomes of peer review can have outsize effects on the careers of participating scholars. As machine learning models are known to amplify biases, we strongly recommend against using the outputs of any machine learning system to make decisions about individual cases. A dataset like DISAPERE is best used to survey participants' behavior. Any interventions based on this information should be subjected to studies in order to ensure that they do not introduce or exacerbate bias.
This material is based upon work supported in part by the National Science Foundation under Grant Numbers IIS-1763618, IIS-1922090, and IIS-1955567, in part by the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C-0106 under Subaward No. 89341790 from the University of Southern California, in part by the Office of Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the University of Southern California, in part by IBM Research AI through the AI Horizons Network, in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction, and in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
Yamen Ajjour, Wei-Fan Chen, Johannes Kiesel, Henning Wachsmuth, and Benno Stein. 2017. Unit segmentation of argumentative texts. In Proceedings of the 4th Workshop on Argument Mining, pages 118-
4We initialize from a sentence-transformers/all-MiniLM-L6-v2 model
128, Copenhagen, Denmark. Association for Computational Linguistics.
Jianzhu Bao, Bin Liang, Jingyi Sun, Yice Zhang, Min Yang, and Ruifeng Xu. 2021. Argument pair extraction with mutual guidance and inter-sentence relation graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3923-3934, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020.
Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000-7011, Online. Association for Computational Linguistics.
Liying Cheng, Tianyu Wu, Lidong Bing, and Luo Si. 2021. Argument pair extraction via attention-guided multi-layer multi-cross encoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6341-6353, Online. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Ruoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2020. Argument mining driven analysis of peerreviews. arXiv preprint arXiv:2012.07743.
Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major NLP conference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1274-1290, Minneapolis, Minnesota. Association for Computational Linguistics.
Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125-179.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2131-2137, Minneapolis, Minnesota. Association for Computational Linguistics.
Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A streamlined method for sourcing discourse-level argumentation annotations from the crowd. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1790-1796, Minneapolis, Minnesota. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
David Soergel, Adam Saunders, and Andrew McCallum. 2013. Open scholarship and peer review: a time for experimentation.
Nandan Thakur, Nils Reimers, Andreas Rückle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.
Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schütze, and Iryna Gurevych. 2020. Fine-grained argument unit recognition and classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9048-9056.
Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021. Can we automate scientific reviewing?
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using
corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguistics.
A Rationale for taxonomy construction
Our label sets leverage ideas from and commonalities between existing work in this domain, including AMPERE (Hua et al., 2019), AMSR (Fromm et al., 2020) ASAP-Review (Yuan et al., 2021), and Gao et al. (2019):
- ASAP-Review's polarity labels approximately correspond to arg-pos and arg-neg labels in AMSR
- AMSR and AMPERE each label non-argumentative sentences in a similar manner
- aspect labels from ASAP-Review apply only to certain types of sentences; namely request and evaluative sentences from AMPERE's taxonomy.
- summary is an exception among ASAP Review's aspects, behaving similarly to AMPERE's quote. We thus include both of these under a structuring category.
Further, in order to gauge the extent to which authors acquiesced to reviewers' requests, we introduce a fine-grained categorization of the types of requests.
- Gao et al. (2019) enumerates some features of rebuttals, including expressing gratitude, promising revisions, and disagreeing with criticisms. We formalize these observations into our rebuttal label taxonomy.
B Annotation tool
Two modes of annotation are possible. First, annotators can apply labels on a sentence-by-sentence basis. Multiple labeling schemas can be annotated simultaneously, with the option of adding constraints so that certain values govern possible values for other properties. This annotation mode is shown in Figure 6.
The second annotation mode can build on the output of the first annotation mode. Here, sentences of a focus text (the rebuttal) are presented in sequence, and annotators are permitted to select one or more of the sentences in the reference text (the review) which form the context of the sentence of the focus text. Further, a label can be applied to the alignment. This annotation mode is shown in Figure 7 and Figure 8.
Figure 6: Review annotation interface. Annotators select label values from dropdown menus for each review sentence in turn. [1] Title of the manuscript whose review is being annotated [2] Reviewer identifier [3] Annotator identifier (removed for anonymity) [4] Link to original forum, in case it is needed for context [5] Individual review sentence [6] Option to report sentence splitting error (sentence splitting generally suffered from false positives) [7] Dropout for REVIEW-ACTION [8] Follow-up dropdownfor FINE-REVIEW-ACTION populated based on value in (7) [9] Button to add second REVIEW-ACTION if necessary (this was seldom used)
C Annotated review-rebuttal pair
Figure 9 shows a truncated version of a reviewrebuttal pair from the train set of DISAPERE.
D Context overlap analysis
As a proxy for agreement of rebuttal spans, we show the types of overlap between spans on rebuttal sentences from 81 examples annotated by two annotators in Table 10.
Type of context overlap	Num. rebuttal sentences	% rebuttal sentences
Exact match	914	53.11%
Partial match	492	28.59%
Agree none	122	7.09%
Disagree none	100	5.81%
No overlap	93	5.40%
Table 10: Types of context overlap. Full agreement is achieved in the top rows (exact match and 'Agree none', where both annotators agree that there is no appropriate subset of review sentences forming the context. in 'Disagree none', one annotator marks a subset of review sentences, while the other does not.
E Additional Agreement Analysis
While some of the IAA scores on annotation are low, we note that the labels used in this task attempt to characterize relatively complex relationships in text. To give more insight into such disagreements, Figure 10 provides a confusion matrix regarding the REBITTAL-ACTION labels. Recognizing that there are often situations in which users of a dataset will hope to reduce a label set, we provide some guidance as to which such merges may be acceptable and which are not.
Many disagreements come from three labels which might be said to exist upon a continuum - ANSWER, MITIGATE CRITICISM and REJECT CRITICISM. We suggest that in the situation of needing to minimize IAA disagreement, one might consider first merging mitigate criticism into reject criticism. The kind of disagreements seen between the two are understandable but nuanced: the difference between saying that the reviewer has a point (but that they disagree on the relevance of that point) and disagreeing with the point itself. Out-of-context rebuttal sentences illustrating this are provided below as examples of this kind of ambiguous situation:
Figure 7: Rebuttal annotation interface. Annotators examine each rebuttal sentence in turn, selecting sentences as context and specifying REBUTTAL-ACTION. [1] Title of the manuscript whose review is being annotated [2] Buttons to navigate between rebuttal sentences. Each page refers to a single rebuttal sentence (See (9)) [3] Reviewer identifier [4] Annotator identifier (removed for anonymity) [5] Link to original forum, in case it is needed for context [6] Link to open pop-up window with full rebuttal text, in case it is needed for context [7] Full review text. When a review sentence is clicked, it is highlighted and its details appear in (11) [8] Option to report sentence splitting error (false positive) [9] Rebuttal sentence being annotated [10] Button to copy REBUTTAL-ACTION label and context from previous rebuttal sentence [11] Table showing details of selected context sentences from the review, with the labels the annotator provided The screenshot is continued in Figure 8.
Figure 8: Rebuttal annotation interface (continued from Figure 7). Annotators examine each rebuttal sentence in turn, selecting sentences as context specifying REBUTTAL-ACTION. [1] Full review text, continued from (7) in Figure 7. [2] Dropout to select context type, in case context cannot be defined as a subset of review sentences. [3] Dropout to select REBUTTAL-ACTION (keyboard navigation possible) [4] Buttons to select REBUTTAL-ACTION (in case mouse navigation is preferred) [5] Option to report egregious sentence splitting errors.
- We note that such rules are indeed limited to some extent, but they still capture a rather expressive fragment of answer set programs with restricted forms of external computations.
- The use of  $C_p^{val}$  for hyperparameter tuning was incidental and not a central point of our paper.
- We agree that the measure theoretic approach is not always necessary (indeed for angular actions, it is not needed), but it is necessary for a very common scenario - clipped actions.
Furthermore, we note that (as illustrated in the confusion matrix) a wide range of disagreements are hard to distinguish from "answer" labels, as authors often attempt to frame disagreements as simple answers to questions.
Figure 9: A (truncated) example from the training set of DISAPERE.
Figure 10: Confusion matrix showing agreement between annotators for REBUTTAL-ACTION labels.