Automatic Analysis of Substantiation in Scientific Peer Reviews
Authors: Yanzhu Guo
Guokan Shang
Virgile Rennard
Michalis Vazirgiannis
Chloé Clavel
Venue: EMNLP 2023 Findings (2023)
Keywords: Peer review
Substantiation
Argument mining
Dataset
TL;DR: We analyse the level of substantiation in scientific peer reviews via an argument mining approach.

Abstract: With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation --- one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence --- and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years. The dataset is available at https://github.com/YanzhuGuo/SubstanReview.

Automatic Analysis of Substantiation in Scientific Peer Reviews
Yanzhu Guo', Guokan Shang', Virgile Rennard 1.4
Michalis Vazirgiannis', Chloé Clavel
LIX, École Polytechnique, Institut Polytechnique de Paris, France 2LTCI, Télécom-Paris, Institut Polytechnique de Paris, France Inria, Paris, France 'Linagora, France Cyanzhu.guo, guokan. hwtpaa/shhannggooytteenniie....e edu vi virgile@rennard. org,
fr chloe. wwwveleteeeeooo-pprra.... fr
Abstract
To this end, our research aims to develop an auto- matic system analyze the quality of peer reviews. Such a system would be of great value not only to chairs, who could use them to eliminate identified poor-quality reviews from the decision-making pro- cess, but also to reviewers, who might be instructed to improve the writing at the time of reviewing. They could be exploited by conference managers as well to analyze overall review quality and re- viewer performance, order better organize the next review round.
With the increasing amount of problematic peer reviews in top Al conferences, the community is urgently in need of automatic quality control measures, In this paper, we restrict our atten- tion to substantiation - one popular quality as- pect indicating whether the claims in review are sufficiently supported by evidence and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as saim-evvdeene pair extraction in scientific peer reviews, and collect Substan- Review, the first annotated dataset for this task. SuvstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argu- ment mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanRe- view dataset obtain meaningful insights on peer reviewing quality in NLP conferences over recent years. The dataset is available at https: //gi thub. com/ Soom//anzhuGu/// BaanhhuGoo/Sustttt Swt//uusttttt tanetan........
Clearly, and according to the review guidelines of several top Al conferences, including ICLR, ICML, NeurIPS, ACL and EMNLP, reviews should be assessed from multiple perspectives (e.g.. do- main knowledgeability. factuality, clarity, compre- hensiveness, and kindness). In this paper, we de- cide focus substantiation, which has appeared in nearly all view guidelines, sometimes under different names such as specificity, objectiveness, or justification. This criterion states that a good review should be based on objective facts and reasoning rather than sentiment and ideology. Specifically, each sub- jective statement (claim) in the review should be backed up by details or justification (evidence). More discussion substantiation will be provided in Section 3.1.
specific quality dimension
1 Introduction
Peer review is an essential practice for gauging the quality and suitability of scientific papers in the academic publication process (Price and Flach, 2017). However, in recent years, this step has been widely criticised for its unreliability, especially in top Artificial Intelligence (AI) conferences (Tran al., 2020). This is partially due to the surge numbers of submitted papers and the shortage of domain experts fulfilling the requirements to serve as reviewers (Publons, 2018; Russo, 2021). This leads to increasing workload per reviewer and paper-vetting by less-experienced researchers (Ghosal et al., 2022), which consequently increases the likelihood of poor-quality reviews. Therefore, developments that can make the quality control process associated with peer reviewing more effi- cient, are likely to be welcomed by the research community (Checco et al., 2021).
To progress towards the goal of automatically evaluating the level of substantiation for any given review, we employ an argument mining approach. Scientific peer reviewing can be viewed as an argu- mentation process (Fromm et al., 2021), in which reviewers convince the program committee of a conference to either accept or reject paper by pro- viding arguments. In our annotation scheme for arguments (see Section 4.2), the two basic com- ponents are claims and evidence. A substantiated argument one where the claims are supported by evidence. Therefore, we formulate the task claim-evidence pair extraction for scientific peer

---

reviews.
A quality aspect highly relevant substantiation, is "justification", previously studied by Yuan et al. (2022) as an evaluation measure for their automatic review generation model. More specifically, they state that "a good review should provide specific reasons for its assessment, particularly whenever it states that the paper is lacking in some aspect". However, their evaluation protocol for justification relies solely human annotators. Our work is the first one automatically assess the substantiation level of scientific peer reviews. More importantly, wc do not only provide a final quantitative score, but also highly interpretable claim Blai--evidence pairs extracted through an argument mining approach.
We release SubstanReview, dataset of 550 peer reviews with paired claim and evidence spans an- notated by domain experts. On the basis of this dataset, we develop an argument mining system to perform the task automatically, achieving satisfac- tory performance. We also propose SubstanScore, metric based on the percentage of supported claims quantify the level of substantiation of each re- view. Finally, we use the SubstanReview dataset to analyze the substantiation patterns in recent con- ferences. Results show a concerning decrease in the level of substantiation of peer reviews in ma- jor NLP conferences over the last few years. Our contributions are threefold:
Argument Mining, Lawrence and Reed (2019) define the task of argument mining the automatic identification and extraction of argument compo- nents and structures. The state-of-the-ar in argu- ment mining was initially based on feature engi- neering, while more recent methods rely on neural networks (Ein-Dor al., 2020), especially follow- ing the introduction of the transformer architecture. Previous works applying argument mining to peer reviews typically focus on identifying argumenta- tive content and classifying it. Hua et al. (2019) introduced the AMPERE dataset containing 400 re- views annotated for proposition segmentation and proposition classification (evaluation, request, fact, reference, quote, or non argument) and trained neu- ral models to perform the two tasks. Similarly, Fromm et al. (2021) performed annotations on 70 reviews for supporting arguments, attacking argu- ments and non arguments, and trained BERT model for the tasks of argumentation detection and stance detection. None of these efforts take into account the structure of arguments, making our work the first to examine the link between different argument components in scientific peer reviews.
1. We define the new task of claim-evidence pair extraction for scientific peer reviews and cre- ate the first annotated dataset for this task.
2. We develop pipeline for performing claim- evidence pair extraction automatically while leveraging state-of-the the-art methods from other well established NLP tasks.
3. We provide meaningful insights into the cur- rent level of substantiation in scientific peer reviews.
2 Related Work
Review Quality Analysis. The exponential growth of paper submissions top Al conferences poses non-negligible challenge to the peer re- viewing process, drawing considerable research attention during recent years (Severin et al., 2022). Multiple peer review datasets have been released along with the corresponding papers, mostly taken from the OpenReview platform (Kang et al., 2018; Cheng et al., 2020; Fromm et al., 2021; Kennard al., 2022). More recent efforts have been made to collect opted-in reviews that are not publicly acces- sible, often through coordination with the program committee (Dycke et al., 2023, 2022).
Task Formulation
In this section, we first take closer look at sub- stantiation and its definition. We then formulate its estimation as the claim-evidence pair extraction task. Finally, we introduce fundamental concepts in argumentation theory and explain how the claim- evidence pair extraction task can be tackled by an argument mining approach.
These resources have been used to carry out studies on peer review quality. Previous works on automatically evaluating scientific peer reviews have targeted the aspects of harshness (Verma et al., 2022), thoroughness (Severin al., 2022), helpful- ness (Severin al., 2022) and comprehensiveness (Yuan al., 2022). Yet, their methodologies are usually based on regression models trained with human annotated scores, which often lack in both generalizability and interpretability.
3.1 Defining Substantiation
While substantiation is only one of the criteria for a good review, is a fundamentally important one.

---

The paper proposes a method to train models for Chinese word segmentation (CWS) on datasets having multiple segmentation criteria.
jective sentiments and no further supporting evi- dence, it is unlikely that the argument provides a fair evaluation that can lead to an appropriate ac- ceptance/rejection decision. Moreover, the purpose of peer reviewing is not only to make the final de- cision but also to provide constructive feedback for the authors to eventually improve their work. This purpose cannot be achieved without sufficient evi- dence substantiating each point made in the review. Compared to other criteria such as factuality, comprehensiveness, or domain knowledgeability, the analysis of substantiation is more straightfor- ward. It does not necessitate a deep understanding of the paper under review. In fact, in our analysis of substantiation, our sole concern is whether each subjective statement has supporting evidence but not whether the supporting pieces of evidence are factually correct. Evaluating the correctness of ev- idence left for future work on the dimension of factuality for peer reviews. However, the annota- tions still need to be carried out by domain experts who have a general understanding of Al research and the context of scientific peer reviews.
Strengths:
1.
[Multi-criteria learning is interesting and
promising-laaimn pos_1
2. [The proposed model is also interest- ing and achieves large improvement from baselines.Jadaim_pos.
Weaknesses:
1. [The proposed method not sufficiently com- pared with other CWS modelss.Jtaamm... _neg____e_g______ [The baseline model (Bi-LSTMM) is proposed [1] and [2]. However, these model is proposed not for CWS but for POS tagging and NE tag- ging. The description "In this paper, we employ the state-of-the--an thrt architecture... ... (in Section 2)
is misleading, Jevidenoc_negeee _eg_1
2. [The purpose of experiments in Section 6.4 is unclear. unclear.Jeioim Jetaim_neg__2 [In Sec. 6.4, the purpose is that investigating "datasets in traditional Chi- nese and simplified Chinese could help each other." However, in the experimental setting, the model is separately trained on simplified Chi- nese and traditional Chinese, and the shared pa- rameters are fixed after training simplified Chinese-Jevidenee_nng What is expected fixed shared parameters?
In short, we define substantiation of scien- tific peer review the percentage of subjective statements that are supported by evidence. There- fore, we propose and formulate the task of claim- evidence pair extraction for scientific peer reviews.
General Discussion:
3.2 Claim-Evidence Pair Extraction
The paper should be more interesting if there are more detailed discussion about the datasets that adversarial multi-criteria learning does not boost the performance.
The task of claim-evidence pair extraction is sep- arated into two steps: claim tagging and evidence linkage. Previous works on proposition segmenta- tion have shown that segmenting peer reviews by sentence boundaries or discourse connectives do not yield optimal results (Hua et al., 2019). There- fore, we do not specify any predefined boundaries for claim or evidence spans. Both steps are per- formed at the token level. An example with anno- tated claim-evidence pairs is shown in Table 1.
[1] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional Istm-erf models for sequence tag- ging. arXiv preprint Stfi:::508.009990
[2] Xuezhe Ma and Eduard Hovy. 2016. End- to-end sequence labeling via bi-directional Istm- cnns-crf. arXiv preprint arXiv: 603.01354.
Table 1: Example of an annotated peer review from the Ruvsta dataset. We select this particular review as an example due to its straightforward organization, where supporting evidence directly follows the claims. However, it should be noted that many other reviews have much more complex structures, rendering the task of claim-evidence pair extraction challenging.
Claim Tagging, The goal of this step is to iden- tify all the subjective statements in a given review. Such statements include evaluation of the novelty, the soundness or the writing of the paper, etc. The definition of subjective statement will be further claborated in Section 4.2. The subjective state- ments are further divided by their polarity. Claims supporting the acceptance of a paper are consid- ered positive while those attacking are considered negative. Therefore, the subtask of claim tagging is formulated as sequence labeling with positive
Most of the times, paper authors are not unwill- ing to accept negative opinions about their work. However, if the argument is only based on sub-

---

and negative types. We adapt the BIO (Beginning. Inside, Outside) encoding scheme, resulting in possible classes for each token (B-claim_ _osiiivv_,ii I- _egaatvv, sodlaim_positive
and commonly exploited in computational linguis- tics settings (Lopes Cardoso et al., 2023).
The main elements of Freeman's 's model are con- clusions and premises. The conclusion is a subjec- tive statement that expresses stance on a certain matter while premises are justifications of the con- clusion. In the context of claim-evidencs pair ex- traction, define claim to be the conclusion and evidence as premise. Freeman (201 la) also pro- poses modality as an argument component indicat- ing the strength of the argumentative reasoning step. Modality is often integrated into the conclusion or not present at all in practical arguments, therefore we do not model it individually but integrate it in the claim span. The last type of argument compo- nent is rehutting/undercuig irrelevant to our analysis of substantiation and are thus not taken into consideration.
and O).
Evidence Linkage. The evidence linkage step follows the claim tagging step. The goal of this step is to select a contiguous span of text from the review as supporting evidence for each re- trieved claim, if such evidence exists. Formally, for each retrieved claim C' = (2..-2,....5 991)) we concatenate it with the full review R (21, ((R) into single sequence S (CLSSccc2....q/// (/ 7 -]m). The task is thus predict the evidence span boundary (start and end token position). We observe the similarity between this task and extractive question answering (QA). In both cases, the goal is to extract the most relevant text span (answer/cvidence), given the context (article/review) and key sentences (ques- tion/claim). We therefore follow the QA model architecture proposed by Devlin al. (2019), pass the concatenated sequence former encoder and train two linear classifiers on top of for predicting the start and end token posi- tion. For claims without supporting evidence, we simply set the answer span to be the special token (CLS]. We make the choice of first tagging claims and then linking piece of evidence to each claim instead of extracting claims/evidence separately and then determining their relations. This way ensure that each piece of evidence is dependent on a claim, since evidence cannot exist alone by definition.
defeaters. They are
Substan Dataset
In this section, we introduce SubstanReview, the first human annotated dataset for the task of claim- evidence pair extraction in scientific peer reviews. We first discuss the source of the reviews and then elaborate on the annotation process. Finally, we provide a statistic overview of our annotated dataset.
pre-trained trans-
4.1 Data Source
The raw reviews used for annotation are taken from the NLPeer dataset (Dycke et al., 2023). NL.eer is the most comprehensive ethically sourced corpus of peer reviews to date. For all included reviews, both the corresponding author and reviewer have agreed to opt in. We use a subpart of NLPeer with reviews issued from NLP conferences (CoNLL 2016, ACL 2017, COLING 2022 and ARR 20221). We deliber- ately choose to only include NLP reviews because conferences in other sub-domains of AI generally vary a lot in review style, which might negatively impact the final system's performance, given the limited amount of total annotation capacity. We randomly select 50% of all available reviews for each of the different conferences, resulting in an annotated dataset of 550 reviews.
3.3 Claim-Evidence Pair Extraction and Argumentation Theory
Argumentation aims to justify opinions by present- ing reasons for claims (Lawrence and Reed, 2019). Scientific peer reviewing can be understood as a process of argumentation where reviewers need to justify their acceptance/rejection recommenda- tions for a paper. We ground the task of claim- evidence pair extraction within the framework of argumentation theory following Freeman's 's model of arguueet (Freeman, 2011b,a). Freeman's 's model integrates Toulmin's model (Toulmin, 2003) and the standard approach (Thomas, 1973). Different from Toulmin's 's model, proposes to analyze ar- guments as product rather than as process. As a result, it is more applicable to real-life arguments
We do not make use of reviews collected without explicit consent through the OpenReview platform, which leads to disregard datasets from previous
I ARR stands for ACL Rolling Review, all reviews included in ARR 2022 are from papers later accepted at ACI. 2022 or NACCL 2022.

---

works (Hua et al., 2019; Fromm et al., 2021). These datasets are not clearly licensed, which might pose ethical and legal issues the long term.
0.8
0.85
0.027
0.051
0.0068
0.069
0.7
0.024
0.031
4.2 Annotation Study
Annotator
0.5
Claim_ neg
0.76
0.16
In this section, we define our annotation scheme, introduce the annotation process and examine the disagreement between different annotators.
0.4
Evidence pes
Annotation Scheme. Our annotation scheme is based on the argumentation model discussed in Section 3.3. The resulting scheme contains the following two argumentative components:
Evidenco_neg
0.1
None
Claim,pe Pas
Claim neg
Eviatence pas
Evidence neg
(1) Claim: Subjective statements that reflect the reviewer's evaluation of the paper. For defining subjectivity, we follow the notion of Wiebe al. (2005): Subjectivity can be expressed either explic- itly by descriptions of the writer's 's mental state or implicitly through opinionated comments. They are further separated by polarity into positive and negative classes.
Annotator
Figure 1: Confusion matrix (normalized by row) be- tween annotations by annotator_ and annotator 2 on reviews labeled by all three annotators (10% of Substan- Review).
this round were only used to refine the annotation guidelines and not included in the final Substan- Review dataset. More results from this annotation round can be found in Appendix B. We believe that the unsatisfactory IAA is due to the high number of annotators which naturally amplifies the bias (Lopes Cardoso et al., 2023). Moreover, the anno- tators only went through 2 hour training session, thus not qualifying as experts for this task. We con- clude that for such complicated annotation tasks involving argumentation theory, is better to em- ploy small number of expert annotators rather than a high number of non experts.
(2) Evidence: Justifications of the subjective statements that serve as claims. For example, the context of reviewer pointing out a weakness the paper, the premise could be specific examples of the problem, reasoning on why this is problematic suggestions for solving the problem.
The complete annotation guidelines can be found in Appendix A.
Inter-annotator Agreement. We use Krippen- dorf's unitizing alpha² (Krippendorff et al., 2016) to calculate ..a-coefficient quantifies the reliability of partition- ing continuum into different types of units. In our case, there are types of units total (positive claims, negative claims, positive evidence, negative evidence and non). The task grows more difficult as the number of types increases.
agreement (IAA). The
Main Round. Following the pilot round, our main annotation study was carried out by three expert annotators who are coauthors of this paper. They are all graduate/post-graduuee NLP researchers pro- ficient in academic English, familiar with the con- text of scientific peer reviews and argumentation theory. They all participated in creating the annota- tion guidelines and supervising the pilot annotation round. They later had several meetings identify factors causing disagreement during the pilot round and further refined the guidelines.
Annotation Rounds. All annotations were done with the open-source data labeling tool doccano We held two rounds total.
Pilot Round. The pilot study was carried out with fourteen annotators following an initial version of annotation guidelines. It was conducted the PeerRead dataset (Kang et al., 2018) since the more comprehensive NLPeer dataset was yet released back then. This round of annotations only led to a moderate a = 0.367. The annotations from
10% of the dataset (55 reviews) was used to calculate inter-annotator agreement (IAA). These 55 reviews were labeled independently by each of the three annotators, resulting in Krippen- dorf's unitizing alpha (Krippendorff et al., 2016) of a 0.657. While other works that simply partition texts into argument and non argument types might achieve even higher IAA, our task is
"https://matie users. eveersggeyc.. "https://doccan http:///cccccoooitubbbb.. tit io/doccano/
fr/agreemett// wuserrsgrrecc.ff/arreeeeer

---

#Claims
%Supported claims
len(Review)
#Reviews
Pos
Neg
All
Pos
Neg
All
CoNLL 2016
2.01
1.94
2.95
27.97
87.03
51.82
483
19
ACL 2017
2.62
2.91
5.54
26.66
78.58
47.72
499
134
COLING 2020
2.70
2.78
5.38
35.04
74.71
45.43
512
56
ARR 2022
2.73
2.25
4.98
30.37
75.54
44.69
472
341
Table Statistics the SubstanReview dataset, reported values are the mean over all reviews. #Claims stands for the number of claims, Suppooree claims stands for the percentage of claims that are paired with evidence.
at a higher difficulty level. Compared to similar ef- forts annotating refined argument structures (Rocha et al., 2022) (ua = 0.33), our IAA score is signifi- cantly improved. The rest 90% of the dataset (495 reviews) was randomly split into three equal portions and each annotated by only one annotator.
4.4 SubstanScore
We propose a quantitative score measuring the over- all substantiation level of given peer review
SubstanScore
Assupported_claing claims x len(review).
As defined in Scction 3.1, a well substantiated review is one where a high proportion of subjec- tive claims are supported by evidence. If review does not contain any subjective claims, sider to be fully objective and assume %sup- ported_claims= claims=100% However, a short review with few or no subjective claims may also con- tain limited substantial information overall, even if "osupported_clan claims is high. To address this bias, we multiply % _aaiddd__mmmm_eee by the re- view length (number of words in the review).
Inter-annotator Disagreement. The token level confusion matrix between annotations (annota- tor tor_l, annotator_22 2) is shown in Figure 1. The confusion matrices between (annotator_i) anno- tator_3) 3) and (annotator_ 2. annotator_3 _3) are highly similar and therefore omitted here. We see that the main disagreement arises between claims and evidence of the same polarity.
4.3 Statistics and Insights
We present several statistics of our annotated Sub- stanReview dataset in Table 2. For all conferences, there exists the same trend that more positive sub- jective claims are detected compared to negative ones. In contrast, the percentage of supported neg- ative claims higher than the percentage of sup- ported positive claims. This is in line with current review practices since most reviewers believe it to be more relevant to provide specific reasoning when stating that paper is lacking in some aspect (Yuan et al., 2022).
During the annotation study, in addition mark- ing the spans, also asked each annotator to the substantiation level of each review on point Likert scale, with 3 representing the strongest level of substantiation.
We calculate the correlation between Sub- stanScore and the human annotated substantia- tion scores. We obtain Spearman's p 0. 7566 (p 6.5 20)), i.e., positive correlation be- tween SubstanScore and human judgements.
We also calculate correlations between Sub- stanScore and %supported _eaiiisslmmm or len(review) separately. Both give worse correlation than the combined SubstanScore.
Conferences included in our analyzed datasets range from 2016 2022. We observe that the av- erage length of reviews are generally on the same level for all NLP conferences, with COLING 2020 the longest and ARR 2022 the shortest. For the proportion of supported claims, there is contin- uous decrease from CoNLL 2016 to ARR 2022. This observation can be understood to correspond the surge of problematic peer reviews in recent years. Our finding is consistent with the one re- ported by Tran al. (2020), that the peer review process has gotten worse over time for machine learning conferences.
5 Experiments and Results
We tackle the claim-evidence pair extraction task formulated in Section 3.2 and construct bench- mark for the SubstanReview dataset. The claim tag- ging treated as a token classification task while the evidence linkage approached as a question- answering task. We solve both of these tasks by fine-tuning pretrained transformer encoders

---

Claim Tagging
Evidence Linkage
Precision
Recall
F1
Exact match
F1
BERT
41.01
52.40
46.01
43.17
78.15
RoBERTa
52.00
59.77
55.61
48.90
80.24
SciBERT
39.66
54.48
45.91
46.69
80.05
SpanBERT 53.67
38.81
36.12
64.31
82.07
Baseline
15.78
9.890
12.16
3,456
10.78
Table 3: Results for the claim-evidence pair extraction task. For the best performing models, we use two-sided I-test to confirm that the results are statistically significant (p < 5%).
(Vaswani et al., 2017), with added task-specific classification layers. To deal with the limited input sequence length of models, split the input re- views into chunks with maximum length of 5 512 tokens (in case longer than the limitation). For evidence linkage, to ensure that the start and end tokens of piece of evidence are in the same chunk, we also add a sliding window with stride of 128.
large version?.
5.2 Experimental Setup
We make a 80/20 split of the dataset for training and testing. Both the train and test splits can be found in the supplementary material. Hyperparam- eters are tuned using 5-fold vross-validation on the training set. Training is done over 10 epochs with batch size of and early stopping. The AdamW W optimizer with 0.01 weight decay is used. Each model is trained 5 times with different randomiza- tion and the mean results are reported. All experi- ments are conducted on two 48GB NVIDIA RTX A6000 GPUs. The average training time is around minutes for the base version model SciBERT and around 11 minutes for the large version models (BERT, RoBERTa and SpanBERT).
5.1 Models
We test the original BERT model along with other alternatives optimized or adapted the do- main/nature of our task:
BERT (Devlin et al., 2019) is the most popular transformer-based model, its base version has 12 encoder layers with 10M parameters in total while its large version has 24 encoder layers with 340M parameters. We use the large version' in our exper- iments.
5.3 Results
In addition the transformer-basee models fine- tuned on SubstanReview, we also compute a base- line for both of the subtasks. For claim tagging, we first segment the reviews into sentences and pass them to sentiment classifier (Barbieri et al., 2020). Tokens in sentences predicted with posi- tive sentiment will be identified positive claims, tokens in sentences predicted with negative senti- ment will be assigned negative claims, tokens predicted with neutral sentiment will be as- signed part of claims. For evidence linkage we select the sentence with the highest BERTScore (Zhang* et al., 2020) similarity to each claim as its evidence. Results are shown in Table 3.
RoBERTa (Liu et al., 2019) is an optimized version of BERT trained with larger dataset using more effective training procedure. We also use the large version of RoBERTa..
SciBERT (Beltagy et al., 2019) uses the original BERT architecture and is pretrained on random sample of 1.14M papers from Semantic Scholar, demonstrating strong performance tasks from the computer science domain. It is released in the base version 6.
SpanBERT (Joshi et al., 2020) modifies the pre- training objectives of BERT to better represent and predict spans of text. It demonstrates strong per- formance tasks relying span-based reasoning such extractive question answering. We use its
Claim Tagging. We report the macro-averaged Precision, Recall and Fl scores for the claim lag-
"https::/huggiingfaac
hnttps://huggingface Spansea/ spanbert-large-cant
'tt hhttss://huggingfacc.com wtwp://n.a......ooreeeeeeeeeeeee "https://huggingfass wwt..../.alle.......a.. iscibrr......t..... scivocab_uncased
/haggsi://uuggingee co/ cardi ffnlp/ twitterr-roberta bastett---b---ee wetaaabassseeeieeeeeeeee sentiment

---

Claim
Evidence
0.8
Pos
Neg
Pos
Neg
None
0.68
0.027
0.078
0.01
0.2
0.7
Precision
78.89
81.34
24.78
75.02
Claim_pos
0.21
0.72
0.0019
0.061
0.0034
0.6
Recall 53.79
67.48
56.79
33.06
True label
0.5
F1 63.78
73.56
34.50
45.23
Cleim_nee nng
0.1
0.022
0.81
0.07
0.4
0.3
Table 4: Combined results of the two subtasks, taking error propagation into account.
Evidence_pee
0.39
0.29
0.056
0.24
0.035
0.2
Evidence_neg
0.13
0.002
0.066
0.025
0.78
0.1
ging subtask. These metries are designed specif- ically evaluate sequence tagging (Ramshaw and Marcus, 1995; Nakayama, 2018), they are very stringent as they only consider a predicted span as true positive if both the class (posi- tive_claing ceee_llaimneegaiveee teeeeiellmm and segmentation ex- actly match the ground truth. The baseline is shown give poor performance, demonstrating the dif- ficulty of this task. Although SciBERT and Span- BERT bring considerable improvements in perfor- mance compared to the original BERT, they are not able to yield as significant of gain as RoBERTa. We thus use the fine-tuned RoBERTa model for claim tagging in our final argument mining system.
0.0
None
Claim, pas
claim neg
Evitence pos
Evidence neg
Predicted label
Figure 2: Confusion matrix (normalized by row) be- tween model predictions and human annotations on the test set (20% of SubstanReview).
extracted than positive ones.
5.4 Error Analysis
The confusion matrix between model predictions and the ground truth shown Figure 2. is simi- lar to the confusion matrix of annotations shown in Figure 1. Both human annotators and the model ap- pear to be struggling to make the same distinctions in the claim-evidence pair extraction task. Error propagation also remains an important challenge evidence tokens are much more often misclassified compared claim tokens. In future work, plan explore a single-step instruction tuning approach to mitigate this problem.
Evidence Linkage. We provide the Exact Match (EM) and FI scores for the evidence linkage sub- task. These the common metrics report for tasks based extractive QA. EM measures the percentage of samples where the predicted evi- dence span identical the correct evidence span, while Fl score is more forgiving and takes into account the word overlap ratio between predicted and ground truth spans. Our models still achieve significant improvement over the baseline. Differ- ent from claim tagging. SpanBERT obtains the best results this subtask. We choose to include the fine-tuned SpanBERT model for evidence linkage in our final system.
5.5 Comparison with Prompt-based Methods
Recently, the widespread recognition and usage of large language models (LLMs) has drawn the paradigm NLP research to prompting methods (Liu et al., 2023). Therefore, we complete our work by providing a case study of using ChatGPT (Ouyang et al.. 2022) to tackle our claim-evidence pair extraction task.
Combined Performance. We analyze the perfor- mance of the whole pipeline, combining the best performing models for each subtask. In Table 4, we show the token level classification results for each class. We observe that the combined pipeline performs better for claims than evidence, despite evidence linkage achieving better results than claim tagging when performed independently. This origi- nates from error propagation, evidence linkage is performed on top of predictions by the claim tagging model instead the ground truth. We also find negative claims and evidence to better
The examples in Appendix demonstrate that ChatGPT is not able achieve satisfactory perfor- mance, with both specifically designed zero-shot and few-shot prompts, which highlights the need of our annotated data for instruction-tune tiriionntttinnnnnnni and the superiority of classical task-specific fine-tuned models proposed in our work.
In future work, our dataset could also be used for
hhttps:://ppnna..om//b//////////

---

instruction tuning and in-context learning, where only small amount of high-quality human curated data required (Zhou al., 2023).
Collecting peer review datasets with more repre- sentative distributions and more diverse domains should be considered for future work.
6 Conclusion
Ethics Statement
this paper, we focus on automatically unalyzing the level of substantiation in scientific peer reviews. We first formulate the task of claim-evidence pair extraction, comprised of two subtasks: claim tag- ging and evidence linkage. We then annotate and release the first dataset for this task: SubstanRe- view. We perform data analysis SubstanReview and show interesting patterns in the substantiation level of peer reviews over recent years. We also define SubstanScore, which positively correlates with human judgements for substantiation. Finally, we develop an argument mining system to auto- matically perform claim-evidence pair extraction and obtain great increase in performance over the baseline approach. We hope that our work in- spires further research on the automatic analysis and evaluation of peer review quality, which is an increasingly important but understudied topic.
All peer review data involved in this paper is used under explicit consent granted by their creators. All datasets published under the CCO CC-BY license. We only use these datasets for research purposes which is consistent with their intended use. While peer review data is highly sensitive and may contain somehow offensive content, the employed datasets are anonymous and cannot be linked to individual people.
During the annotation procedure, have noti- fied the annotators of the intended use of the dataset and obtained their full consent to publish the anno- tations. All annotators are paid fair wage for their efforts.
We would like to state that the argument mining system resulting from our project is still research prototype and should not be utilized for practical evaluations, especially when decision making is involved. Its validity and fairness still need be extensively tested in real-world settings.
Limitations
The main limitation of our work lies in the re- stricted scope of peer reviews included in our dataset.
A Acknowledgements
We thank the ANR-TSIA HELAS chair for sup- porting the first and fourth authors.
Although attracting increasing amounts of at- tention recently, clearly licensed datasets of peer reviews are still very scarce and all of them are based on donation-based workflow. This means that we only have access to peer reviews of which both the paper author and reviewer have expressed explicit consent to opt in. This introduces bias in our dataset, as reviewers are more likely to give consent to publish their reviews if they are confi- dent in the quality of the review. Therefore, the quality of reviews (including the level of substanti- ation) included in our annotated dataset might be skewed towards the higher end. Systems trained on these data may encounter problems when applied in world scenario, where the review qualities are more balanced.
We also thank the annotators who participated in the pilot round of annotation study: Hadi Ab- dine, Johannes Lutzeyer, Ashraf Ghiye, Christos Xypolopoulos, Ayman Qabel, Moussa Kamal Ed- dine. Iakovos Evdaimon, Sissy Kosma, Yassine Abbahaddou, Giannis Nikolentzos and Michalis Chatzianastasis.
References
Francesco Barbieri, Jose Camacho-Collados. Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644 1650, Online. Association for Computational Linguistics.
Given the high level of expertise required and significant amounts of efforts involved to annotate peer reviews, we could not perform the annotations on larger scale. Thus, dataset to only include peer reviews from NLP conferences, leading to the potential lack of domain generalizability of our argument mining system.
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019, SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIPP pages 3615-
have restricted our

---

3620, Hong Kong, China. Association for Computa-
Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Plos one, 17())c0225223..
tional Linguistics.
Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. Ai-assisted peer review. Humanities and social sci- ences communications, 8(1):1-11.
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and
Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2131 2131-2137, Minneapolis, Minnesota. Association for Computational Linguistics.
Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000-7011, Online, Association for Computa- tional Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Danicl S. Weld, Luke Zeutlemoee, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguisties. 8:64-77.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Pupers), pages 1647-1661, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. Yes-yes-yes: Proactive data collection for ACL rolling review and beyond. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 300-318, 18, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Neha Kennard, Tim O'Gorman, Rajarshi Das, Ak- shay Sharma, Chhandak Bagchi, Mathew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and An- drew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1234-1249, Seattle, United States. Association for Computational Linguistics.
Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023, NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 5049- 5073, Toronto, Canada. Association for Computa- tional Linguistics.
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Mar- tin Gleize, Leshem Choshen, Yufang Hou, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2020. Cor- pus wide argument mining-a working solution. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 34(05)).683-7663 -6911
Klaus Krippendorff, Yann Mathet, Stéphane Bouvry, and Antoine Widlöcher. 2016. On the reliability of unitizing textual continua: Further developments. Quality & Quantity, 50(6):2347-233:
John Lawrence and Chris Reed. 2019. Argument min- ing: A survey. Computational Linguistics, 45(4):765- 818.
James B Freeman. 201 la. Argument Structure:: Repre- sentation and Theory, volume 18. Springer Science Business Media.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
James B Freeman. 2011b. Dialectics and the macrostructure of arguments. In Dialectics and the Macrostructure of Arguments. De Gruyter Mouton.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 197.1199...1162..
Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Ruoxia Qi. Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2021. Argument mining driven analysis of peer- reviews. Proceedings of the AAAI Conference on Artificial Intelligence, 3566)47588-776.
Henrique Lopes Cardoso, Rui Sousa-Silva, Paula Car- valho, and Bruno Martins. 2023. Argumentation

---

models and their use corpus annotation: Practice, prospects, and challenges. Natural Language Engi- neering, page 1-38.
Chapter of the Association for Computational Lin- guistics and the 12th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 925-935, Online only. Association for Computational Linguistics.
Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from http:://ithhu..co////
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation. 39(2):165-210)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Newral Information Processing Systems, 35:27730-27744,
Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212.
Simon Price and Peter A Flach. 2017. Computational support for academic peer review: A perspective from artificial intelligence. Communications of ACM, 60-3):70-79.
Tianyi Zhang*, Varsha Kishore Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Publons. 2018. Global state of peer review 2018.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305. 1206.
Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation-base learning. In Third Workshop on Very Large Corpora.
Gil Rocha, Luis Trigo, Henrique Lopes Cardoso, Rui Sousa-Silva, Paula Carvalho, Bruno Martins, and Miguel Won. 2022. Annotating arguments in cor- pus of opinion articles. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 1890-1899, Marseille, France. European Language Resources Association.
Alessio Russo. 2021. Some ethical issues in the re- view process of machine learning conferences. arXiv preprint (1X.:::2160000000
Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vilstrup Mouatt, and Stefan Müller. 2022. Journal impact factor and peer review thoroughness and helpfulness: su- pervised machine learning study. arXiv preprint arXiv:2207.09821.
Stephen Naylor Thomas. 1973. Practical reasoning in natural language.
Stephen Toulmin. 2003. The of argument. Cam- bridge university press.
David Tran, Alex Valtchanov, Keshav Ganapathy, Ray- mond Feng, Fric Slud, Micah Goldblum, and Tom Goldstein. 2020. An open review of openreview: A critical analysis
machine learning conference review process. arXiv preprint arXiv:2010.05137
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Rajeev Verma, Rajarshi Raychoudhury, and Tirthankar Ghosal. 2022. The lack of theory is painful: Mod- eling harshness in peer review comments. In Pro- ceedings of the 2nd Conference of the Asia-Pacific

---

A Annotation Guidelines
Annotating Peer Reviews to Evaluate Substantiation
With the increasing amount of problematic peer reviews in top Al conferences, the community is urgently in need of automatic quality control measures. The goal of our project is to evaluate the substantiation scientific peer reviews. We aim to develop an argument mining system that can automatically detect claims that are vague, generic and unsubstantiated. For this purpose, we ask all annotators to participate in the creation of the SuvstanReview dataset, which will eventually be publicly released for research purposes. The task is to highlight claims in reviews from the NLPeer dataset, as well as the evidence of each claim (if they exist).
[Freeman's 's model of argument]
Premise
UCCA representation derivel inguidic untion syntax but is sill a to construct
Defeaters (not annotated)
Modality
Claim
abstract symbolic representation compositional symbollo ropr rssantation bood procisoly grammar.
(So, presuumntyy
iiss incomeet for the authors In refer their model as "grammartess"
(Uinleoe Fan withars have a dffeernt delintion of grammar)
Evidence
Figure 3: An example of the structure of an argument extracted from a review annotated according to Freeman's 's model of argument.
The main elements of Freeman's 's model are conclusions and premises. The conclusion is a subjective statement that expresses a stance on a certain matter while premises are justifications of the conclusion. In the context of claim-evidence pair extraction, we define claim to be the conclusion and evidence premise. Freeman also proposes modality as argument component indicating the strength of the urgumentative reasoning step. Modality is often integrated into the conclusion not present at all in practical arguments, therefore we do not model it individually but let it take part in the claim span. The last type of argument component is rebutiingundercutiing defeaters. They are irrelevant our analysis of substantiation and thus not taken into consideration.
[Major Claims vs. Claims]
We model the argumentation structure of a review as a two-level tree structure. The major claim (level 0) is the root node and represents the reviewer's general standpoint on whether the paper should be accepted. The major claim should not be annotated. We aim at annotating the more specific arguments which either support or attack the major claim. These arguments are further separated into claims (level 1) and evidence (level 2).
[aak overview]
1. Annotate Claims (Def: Subjective statements that convey the reviewer's evaluation of the research, related the paper acceptance/rejeccion decision-makiin process).
Claims are characterized by their subjectivity. Subjectivity can be expressed explicitly by de- scriptions of the writer's mental state, such as in "I'm not convinced of many of the technical details in the paper" " and in I disagree a little bit here ". It can also be expressed implicitly by opinionated descriptions the work as in There is no clear definition given for what this means

---

and " This paper lacks quite a bit comparison with existing work ". Both the adjective "clear" and the verb "lack" indicate subjective opinions and need to be further justified.
Two evaluations should be separated if they evaluating different aspects of the paper.
Evaluations should be numbered according to their order of appearance.
2. Annotate Evidence (Def: Justifications of the subjective statements that serve as claims, For example, the context of a reviewer pointing out weakness of the paper, the premise could be specific examples the problem, reasoning on why this problematic or suggestions for solving the problem.).
Not all claims have an evidence.
Evidence do not have to be correct.
Evidence can appear both before and after the Claim.
Evidence should be numbered according to the claim they support.
3. In the comments section of each review
Strength
Click to comment on document
Figure 4: Comment button in the toolbar on the top of the user interface.
Rate the substantiation of each review scale from to 5.
(Poor) The vast majority of the claims in the review vague, generic, and unsubstantiated. No comments that specifically relate the content/substance of the work.
(Insufficient) Few of the claims in the review are substantiated, i.e., supported by evidence. (Average) - The review contains both valid and supported claims as well as some unsubstantiated
statements and opinions.
(Sufficient) - Most important claims are well justified although some claims and opinions require further substantiation.
(Solid) The vast majority of the claims are meaningful and well supported with evidence; reviewer's opinions well argued for.
Rate the annotation difficulty of each review on a scale from I to 3. (easy); 2 (medium); 3 (difficult)
Write the substantiation score first and the annotation difficulty score second.
The two scores should be separated by semicolon (:)
Comments
Q22
Figure 5: An example of comment with scores separated by semicolon.
[Further clarifications]
1. Annotations are done at the token level.

---

You can annotate arbitrary spans of text without taking into consideration any punctuation marks.
Favor longer spans of text whenever possible, do not only annotate keywords.
2. Claims and evidence spans can coexist within the same sentence. Example: "However, the major limitation the reviewer captures from the paper ((vaaluaiion) is that
the BCD is only used during the test stage I(justification)"
3. A lot of content in the text might remain unannotated. For example, facts that do not justify an explicit evaluation should not be annotated. Reactions to rebuttals should also not be annotated.
4. Don't 't forget to click on the leftmost button in the toolbar to mark that an annotation is completed.
Not Checked gths:
Figure 6: Check button in the toolbar the top of the user interface.
B Pilot Annotation Round
We introduce pilot round of annotation. Although it only led to moderate IAA (, = 0.367), it helped better understand the underlying difficulties and improve our annotation methodology accordingly. Annotation Process. All annotators have gone through a 2 hour training session before proceeding with the annotations. Each annotator also asked to complete three annotation samples individually and given feedback on their performance, verifying that they have understood correctly the annotation principles. Each review is randomly assigned to three annotators resulting in an average of 67 reviews per annotator. Annotators report an average of 4 hours to complete the assigned task. These hours count as normal working hours under their research contracts and are paid well above the local minimum wage. However, this round of annotations only led to moderate annotator agreement (IAA) with Krippendorf's unitizing alpha a 0.367.
Post-processing. To aggregate annotations from different annotators into the final dataset, post- processing step required. We build consensus between different annotators and obtain a unique annotated span for each claim and evidence.
For annotations of claims, we assign a label to a token if at least two of the three annotators have chosen the same label (majority voting).
The final evidence annotations are built upon the aggregated set of claim annotations. To solve the problem that the aggregated claims may not exactly match the claims annotated by each annotator, we examine the percentage of word overlap between them. For each aggregated claim C₁ in the final dataset, if claim C% (annotated by annotator we consider to correspond C, The evidence E linked to C7 by annotator is thus also considered linked to the claim C. Just like with claim aggregation, all the evidence spans linked to C, are aggregated via majority voting.
(1, 3)) has at least 60% percent word overlap with C₁ then
Dataset Statistics. We present several statistics in Figure 7. The annotated dataset contains 314 reviews with an average length of 518 words. Comparing between claims and evidence, we observe that the average length of evidence (35) is significantly longer than that of claims (13). When comparing positive and negative classes, find that the average number of negative claims per review (1.80) is slightly higher than positive ones (1.36) and that the average length of negative claims (14) longer than that of positive claims (11). The average length of evidence for both negative and positive claims is almost the same (30 and 29 respectively). However, the range of length for negative evidence much wider (up 100). The percentage of supported negative claims is 84.91 % while is only 61.24% for positive ones. As a general trend, reviewers tend to provide more detailed explanations for their negative evaluations.

---

words
75
#
400
#claims
10
desupport claims
2000
50
200
0.5
1000
Reviews (a) len(review)
Major Pos Neg
Pos
Neg
Major Pos Neg
Pos
Neg
(b) len(claim)
(c) len(evidence)
(d) #claims
(e)
Ssupported claims
Figure 7: Box plot of statistics for the SubstanReview dataset. Orange lines represent the median, green triangles represent the mean.
C Case study on ChatGPT
In this section, conduct case study using ChatGPT (May 24, 2023 version) through the platform provided by OpenAL. We provide multiple examples for analyzing the substantiation level of the peer review in Table 5 with different prompting techniques. Zero-shot prompting.
this case, ChatGPT is directly inquired to deal with the tasks claim extraction and evidence linkage, without providing any prior information, results can be found in Table and Table 7, respectively. For the claim extraction task, we observe that ChatGPT cannot distinguish between subjective claims and evidence. It mistakes the facts supporting the claim as also being claims.
For the evidence linkage task, ChatGPT fails completely. It only repeats the claim again, adding along some irrelevant information. This might be connected its inability to distinguish between claims and evidence in the first place.
Zero-shot prompting with task descriptions.
this case, we additionally provide the task descriptions in our prompts, more specifically, the definition section of claim and evidence from our annotation guideline (see Table 12).
Results in Table 8 show that ChatGPT achieved a much better performance on the claim extraction task than the previous zero-shot case, all negative claims are correctly extracted.
Results in Table 9 demonstrate that despite having comprehensive task descriptions, the process of evidence linkage continues pose significant challenges. In the output list of evidence, only the first is relevant the claim of interest.
Few-shot prompting with task descriptions.
this case, on the basis the previous prompts, we give ChatGPT more information by providing expected claim and evidence extractions for one example review (see Table 13).
Based on the data presented in Table 10 and Table 11, we can deduce conclusions similar to the prior case. It's evident that ChatGPT struggles to differentiate between claims and evidence and to denote their relationships.
The above examples demonstrate that ChatGPT is not able to achieve a satisfactory performance, with both specifically designed zero-shot and few-shot prompts, which highlights the need of our annotated data for instruction-tuning and the persistent superiority of classical task-specific fine-tuned model proposed in our work.

---

[Review 1]
The paper proposes method train models for Chinese word segmentation (CWS) on datasets having multiple segmentation criteria.
Strengths:
Multi-criteria learning is interesting and promising.
2. The proposed model is also interesting and achieves a large improvement from baselines.
Weaknnsses:
1. The proposed method is not sufficiently compared with other CWS models.
The bascline model (Bi-LSTM) is proposed in [1] and Howeer,, these model is proposed not for CWS but for POS tagging and NE tagging. The description "In this paper, we employ the state-of-the architecture... ... (in Section 2) is misleading.
2. The purpose experiments Section 6.4 is unclear. In Sec. 6.4, the purpose is that investigating "datascts in traditional Chinese and simplified Chinese could help each other." However, in the experimental setting, the model is separately trained on simplified Chinese and traditional Chinese, and the shared parameters are fixed after training on simplified Chinese.
General Discussion:
The paper should be more interesting if there are more detailed discussion about the datasets that adversarial multi-criteria learning does not boost the performance.
[1] Zhiheng Huang, Wei Xu, and Kai Yu. 2015, Bidirectional Istm-crf models for sequence tagging. arxiv preprint
[2] Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional Istm-cnns- 10X.01603.01354 603.01354.
crf. Xiv preprint arXiv:
Table 5: Example review for case study.
Extract the negative subjective claims from the following review:
[Review 1]
Output:
Subjective negative claims:
"The proposed method is not sufficiently compared with other CWS models."
2. "The baseline model (Bi-LSTM) is proposed in [1] and [2]. However, these model is proposed not for CWS but for POS tagging and NE tagging."
3. "The description 'In this paper, employ the state-of-the-art architecture... (in Section 2) is misleading."
4. "The purpose of experiments in Section 6.4 is unclear."
5. "In Sec. 6.4, the purpose is that investigating 'datasets in traditional Chinese and simplified Chinese could help each other.' However, in the experimental setting, the model separately trained on simplified Chinese and traditional Chinese, and the shared parameters are fixed after training on simplified Chinese."
6. "The paper should be more interesting if there are more detailed discussion about the datasets that adversarial multi-criteria learning does not boost the performance."
Table 6: An example of ChatGPT performing the claim extraction task (zero-shot prompting).

---

Extract the evidence for the claim
"The proposed method is not sufficiently compared with other CWS models."
from the following review:
[Review 1]
Output:
The evidence for the claim "The proposed method not sufficiently compared with other CWS models" in this review the statement made by the reviewer that "The proposed method is not sufficiently compared with other CWS models." TThi claim is presented as a weakness of the paper, indicating that the reviewer believes that the paper's ' lack of comparison to other CWS models is negative aspect of the research.
Table 7: An example of ChatGPT performing the evidence linkage task (zero-shot prompting).
You will be given a guideline for annotating / extracting subjective claims and their relative evidence (as text spans) in scientific peer reviews, to evaluate the level of substantiation. Annotation guideline: [Guideline]
Extract the negative subjective claims only (not evidence) from the following review:
[Review 1]
Output:
Negative subjective claims:
1. The proposed method is not sufficiently compared with other CWS models.
2. The description "In this paper, we employ the statc-of-the architecture ..." (in Section 2) is misleading.
3. The purpose of experiments in Section 6.4 is unclear.
4. The paper should be more interesting if there more detailed discussions about the datasets that adversarial multi-criteria learning does not boost the performance.
Table example of ChatGPT performing the claim extraction task (zero-shot prompting with task descriptions).
You will be given guideline for annotating / extracting subjective claims and their relative evidence (as text spans) in scientific peer reviews, to evaluate the level of substantiation. Annotation guideline: [Guideline]
Extract the evidence for the claim
"The proposed method is not sufficiently compared with other CWS models." "
from the following review:
[Review 1]
Output:
Claim: 1. The proposed method is not sufficiently compared with other CWS models.
Evidence:
The baseline model (Bi-LSTM) is proposed in [1] and [2]. However, these models are proposed not for CWS but for POS tagging and NE tagging.
2. The description "In this paper, we employ the state-of-the-art architecture ..." (in Section 2) is misleading.
3. The purpose of experiments in Section 6.4 is unclear. In Sec. 6.4, the purpose is investigating "datasets in traditional Chinese and simplified Chinese could help each other." However, in the experimental setting, the model is separately trained on simplified Chinese and traditional Chinese, and the shared parameters are fixed after training on simplified Chinese.
Table 9: An example of ChatGPT performing the evidence linkage task (zero-shot prompting with task descriptions).

---

You will be given guideline for annotating / extracting subjective claims and their relative evidence (as text spans) in scientific peer reviews, to evaluate the level of substantiation. Annotation guideline: [Guideline]
Annotation example for one review:
[Example Review]
For the above example review, the extracted claims and evidence are:
Positive claim 1: [Example Claim 1J Evidence I: [Example Evidence 1]
Negative claim 2: [Example Claim 2] Evidence 2: [Example Evidence 2]
Negative claim 3: [Example Claim 3] Evidence 3: [Example Evidence 3]
Extract the negative subjective claims only (not evidence) from the following review:
[Review 1]
Output:
Negative claim 1: The proposed method is not sufficiently compared with other CWS models.
Negative claim 2: The purpose of experiments in Section 6.4 is unclear.
General negative claim 3: The paper should be more interesting if there more detailed discussion
about the datasets that adversarial multi-criteria learning does not boost the performance.
Table 10: An example of ChatGPT performing the claim extraction task (few-shot prompting with task descriptions).
You will be given a guideline for annotating / extracting subjective claims and their relative evidence (as text spans) in scientific peer reviews, to evaluate the level of substantiation. Annotation guideline: [Guideline]
Annotation example for one review:
[Example Review]
For the above example review, the extracted claims and evidence are:
Positive claim [Example Claim 1] Evidence [Example Evidence 1]
Negative claim [Example Claim 2] Evidence [Example Evidence 2]
Negative claim 3: [Example Claim 3] Evidence 3: [Example Evidence 3]
Extract the evidence for the claim
"The proposed method is not sufficiently compared with other CWS models."
from the following review:
[Review 1]
Output:
Claim I: The proposed method is not sufficiently compared with other CWS models.
Evidence 1: The baseline model (Bi-LSTM) is proposed [1] and [2]. However, these models are
proposed not for CWS but for POS tagging and NE tagging. Evidence 2: The description "In this paper, we employ the state-of-the-an architecture ..." (in Section 2) is misleading.
Table 11: An example of ChatGPT performing the evidence linkage task (few-shot prompting with task descriptions).

---

[Guideline]
1. Annotate claims (Def: the reviewer's evaluation of the research, related to the paper accep- tance/rejection decision-making process).
We only annotate claims that contain subjective judgements.
Subjectivity can be expressed explicitly by descriptions of the writer's mental state, such as in " I' m not convinced of many of the technical details the paper" and in disagree a little bit here". It can also be expressed implicitly by opinionated descriptions of the work as in "There is no clear definition given for what this means" and "This paper lacks quite bit in comparison with existing work". Both the adjective "clear" and the verb "lack" indicate subjective opinions and need to be further justified.
Two claims should be separated if they are evaluating different aspects of the paper.
Claims should be numbered according to their order of appearance.
2. Annotate evidence (Def: groundswwarratbbbkiiggguuuuii that the reviewer expressed to support
the above claim).
Not all claims have a premise.
Evideecee do not have to be correct.
EEvidence can appear both before and after the evaluation.
EEvidence should be numbered according to the evaluation they support.
Table 12: Guideline for prompts.
[Example Review]
summary_ of_strengths
How to deal with negation semantic is one of the most fundamental and important issues in NLU, which is especially often ignored by existing models. This paper verifies the significance of the problem on multiple datasets, and in particular, proposes to divide the negations into important and unimportant types and analyzes them (Table 2). The work of the paper is comprehensive and solid. summary_or weaknesses
However, I think the innovation of this paper is general. The influence of negation expressions NLP/NUU tasks has been widely proposed in many specialized studies, well as in the case/error analysis of many NLP/NLU tasks. my opinion, this paper is the only integration of these points of view and does not provide deeper insights to inspire audiences in related fields. [Example Claim 1]
The work the paper is comprehensive and solid.
[Example Evidence 1]
This paper verifies the significance of the problem on multiple datasets, and in particular, proposes to divide the negations into important and unimportant types and analyzes them (Table 2).
[Example Claim 2]
However, think the innovation of this paper is general.
[Example Evidence 2]
The influence of negation expressions on NLP/NLU tasks has been widely proposed many specialized studies, as well as in the case/error analysis of many NLP/NLU tasks.
[Example Claim 3]
does not provide deeper insights to inspire audiences in related fields
[Example Evidence 3]
this paper is the only integration of these points of view
Table 13: Example review, claims, and evidence for prompts.


Reviews:
[{"review_id": "EkFICavhPV", "reviewer": "Reviewer_td1b", "summary": "", "questions": "1. Check the reasons to rejct\n2. Table 2 seems difficult to understand. What are the total no of positive/negative claims rather than % ?\n3.  Table 3 for evidence linkage can you also report Precisn, Recall?\n4. For table 3 and 4 since the dataset is annotated by different annotators, it is difficult to understand what is the upper bound of performance. Can you justify this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the motivation, writing and analysis of our paper. We would like to resolve several misunderstandings, especially regarding the annotation process. We reply to your comments below.\n\n**Response to “Reasons to reject”:**\n1. We disagree on the point that the quality of our dataset is low as a result of 10% of reviews being used to estimate the IAA. Firstly, in the only prior work in which peer reviews were annotated at the token level [1], IAA was estimated on the basis of the entire data set containing only 77 reviews. 10% of our comparatively large dataset, containing 550 reviews, is therefore of the same order of magnitude as previously published work. We furthermore want to mention that reviews from the different conferences are proportionally represented in our 10% sample. This stratification according to the originating conference allows us to use our sample to estimate the IAA across the data set as accurately as possible. \n\n2. There exists bias in all human annotations, it is only possible to mitigate it but not completely avoid it. While we cannot claim that there is no bias in our dataset, we do not believe that including expert annotators as co authors introduces any additional bias. In fact, it is ethical to include them as co authors since one of the key contributions of the paper is the dataset which they created. The annotation guidelines were initially established by the main author of the paper, the expert annotators participated in making adjustments through providing feedback and suggestions. Their involvement helps to identify potential areas of ambiguity and reduces the chance of misinterpretation. This is recommended practice for improving annotation quality [2].\n\n3. This is a misunderstanding. The quoted sentence is from the description of the pilot annotation round. The annotations from this round were not included in the final SubstanReview dataset (as written in Lines 376 –377). We include this section only for completeness and to prove that a small number of expert annotators are much better than a large number of non-experts for such a complicated task. We will further emphasize this in the final version of our paper. The expert annotators who created SubstanReview have gone through approximately 20 hours of training/discussion spread over a month before proceeding to the final annotations. We will also add this to the revised manuscript.\n\n4. Thank you for bringing this to our attention. We will make sure to also mention SubstanScore in the Introduction and Conclusion sections of our revised manuscript. Feel free to let us know if you believe further discussion of the score in other parts of the paper to be beneficial.\n\n**Response to “Questions for the authors”:**\n1. Please refer to previous responses.\n\n2. The total number of positive/negative claims are reported in the second and third columns of Table 2. **#Claims** stands for the *number of claims*, **%Supported claims** stands for the *percentage of claims that are paired with evidence*. **Pos** means *positive* and **Neg** means *negative*. In the final version of the paper, we will add these explanations to the caption.\n\n3. The evidence linkage step is treated as an extractive question answering task, the common metrics to report for this task are Exact Match and F1. However, we agree that we should add the precision and recall scores for consistency. Please see the requested metrics displayed in the table below.\n\n|          \t| Precision \t| Recall \t|\n|----------|-----------|--------|\n| BERT     \t| 75.99     \t| 80.48  \t|\n| RoBERTa  \t| 79.48     \t| 81.03  \t|\n| SciBERT  \t| 79.09     \t| 81.03  \t|\n| SpanBERT \t| 82.39     \t| 81.77  \t|\n| Baseline \t| 12.19     \t| 9.67   \t|\n\n4. This is another misunderstanding. Table 3 and 4 result from the same annotations (the main round). The difference between the two tables is that Table 4 takes into account error propagation while Table 3 does not. The two subtasks (claim tagging and evidence linkage) are consecutive. In Table 3, the evidence linkage step is performed on ground truth claims, whereas in Table 4, evidence linkage is performed on top of claims tagged by the system. As expected, the performance in Table 3 is better than that in Table 4.\n  \n[1] Fromm, Michael, et al. \"Argument mining driven analysis of peer-reviews.\" AAAI 2021.\n\n[2] Klie, Jan-Christoph, Richard Eckart de Castilho, and Iryna Gurevych. \"Analyzing Dataset Annotation Quality Management in the Wild.\" arXiv preprint arXiv:2307.08153(2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper address one of the important issues in peer reviewing, the substantiation or specificity of reviews.\n2. A new dataset and baselines are proposed for the task\n3. Sufficient amount of analysis has been performed and the insights are interesting.", "reasons_to_reject": "1. the dataset developed in the process does not seem to be very high quality. \"10% of the dataset (55 reviews) was used to\ncalculate inter-annotator agreement (IAA)\" seems too low a number to judge the quality of the dataset.\n2. \"Following the pilot round, our main annotation study was carried out by three expert annotators who are coauthors of this paper. They are all graduate/post-graduate NLP researchers\". The annotators being the authors of the paper who created the guidelines seem to bring in a lot of bias in the dataset.\n3. \"Moreover, the annotators only went through a 2-hour training session, thus not qualifying as experts for this task\"- The training time seems too less. \n3. The substantiation score introduced in the paper is never discussed further in the paper other than Sec 4.4", "questions_to_authors": "1. Check the reasons to rejct\n2. Table 2 seems difficult to understand. What are the total no of positive/negative claims rather than % ?\n3.  Table 3 for evidence linkage can you also report Precisn, Recall?\n4. For table 3 and 4 since the dataset is annotated by different annotators, it is difficult to understand what is the upper bound of performance. Can you justify this?", "paper_topic_and_main_contributions": "The paper studies substantiation which is one of the important aspects in the domain of peer reviews. The paper models the problem as a claim-evidence extraction task. The paper is easy to follow. A new dataset for this task is proposed and argument mining systems developed to address this task.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 338-348 too long sentences, and Line 253 incomplete\n2. Line 286 no citation for Freeman"}, "extra_text": {}}, {"review_id": "bi8Ib4kPP5", "reviewer": "Reviewer_JsBA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work. We are glad that you found our annotated dataset to be valuable, the developed argument mining system to be useful, and the results of our analysis to be interesting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An expert-annotated dataset of 550 peer reviews with paired claim and evidence spans. While useful for the specific task framed by the paper, this dataset could potentially be used by the broader argument mining research area. \n- Data analysis of peer reviews from 2016 to 2022. The decrease in supported claims in peer reviews is a very interesting result (though not new/original, previous work has shown similar results, as stated by the authors). \n- Argument mining system usefully adapted to the problem at hand, framed as two consecutive tasks: claim tagging (token classification task) and evidence linkage (question answering task)\n", "reasons_to_reject": "- No specific reason to reject. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about automatically assigning the level of substantiation of a peer review, i.e determining whether the claims in a review are sufficiently supported by evidence. To do so, the authors formulate the task as claim-evidence pair extraction from the peer review.  The task consists of two steps: claim tagging (claim detection with binary sentiment label), and evidence linkage (for each detected claim in the previous step, retrieve the contiguous span of text from the review which supports each claim).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jsYVXOCTGK", "reviewer": "Reviewer_M4ST", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work. We are glad that you found our dataset to be useful, approach to be novel/interesting and experiments to be sound. Regarding the size of the dataset, we agree that it is not large in the sense of today’s “big data era”. However, our experiments have proven that our dataset is efficient for the claim-evidence pair extraction task. We focus on quality rather than quantity, our dataset is very carefully curated. In future work, our dataset could also be used for instruction tuning and in-context learning, where only a small amount of high-quality human curated data is required. We will make sure to add this point to our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Useful dataset (though not very large)\n- Interesting and relatively novel approach based on argument mining, sound experiments", "reasons_to_reject": "- not a major reason for rejection, but the relatively small size of the dataset is the biggest weakness", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a study of automatic claim substantiation evaluation in scientific reviews, and introduces a dataset of scientific reviews annotated by marking spans of text containing claims and evidence. They also propose a method for automatically pairing claims and evidence on this dataset based on an argument mining framework.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 56: \"domain knowledgeability\"  could be simply \"domain knowledge\"?\n- "}, "extra_text": {}}, {"review_id": "BUwkjC5Zi3", "reviewer": "Reviewer_7zci", "summary": "", "questions": "A. Could you provide any justification for using a sentiment classifier based on tweets for the baseline?\n\nB. How do you plan to address the potentially limited generalizability of your approach and findings? Have you considered any strategies to adapt or validate your approach on reviews from other research domains to ensure broader applicability?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. We are extremely glad that you acknowledge the careful curation of our dataset and find it to be valuable despite its limitation in scope. This exactly corresponds to our considerations in the creation of the dataset. We further reply to your suggestions in detail.\n\n* We agree that the third claim on the contribution should be refined. We will change it to “We provide meaningful insights into the current level of substantiation in publicly available peer reviews from recent NLP conferences.” \n\n* We will clarify that the models used in the experiments are identified by hugging face tags.\n\n* **Question A**:\nIn fact, sentiment classification for peer review sentences is a simple task because of the explicit expressions of opinions. As demonstrated by the confusion matrix in Figure 2, positive and negative classes are very rarely confused. This task is thus unlikely to be a bottleneck in the baseline’s performance. To the best of our knowledge, there does not exist an open-source sentiment classifier trained on scientific text. Therefore, our choice to make use of a high-performing open-source sentiment classifier trained on tweets [1], appears to be suitable to us and potential bias introduced by the training on tweets should be of limited impact.\n\n* **Question B**:\nWe believe that the formulation of the claim-evidence pair extraction task is generalizable to other research domains since the underlying argumentation theory is shared. However, the models would need to be further fine-tuned with domain-specific annotations since the reviewing format and terminology could be very different. For future work, we could extend our annotations to the F1000-22 subpart of the NLPeer dataset [2] which covers a wide range of research communities from scientific policy research to medicine and public health. This would require us to collaborate with domain experts from the corresponding fields. \n\n[1] Barbieri, Francesco, et al. \"TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification.\" Findings of EMNLP 2020.\n\n[2] Dycke, Nils, Ilia Kuznetsov, and Iryna Gurevych. \"NLPEER: A Unified Resource for the Computational Study of Peer Review.\" ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new dataset which is the first annotated dataset for this task. This dataset has been carefully curated and annotated by domain experts, making it a valuable resource for the NLP community. \n2. The paper provides a clear and detailed description of the data collection guidelines used to create the SubstanReview dataset. Additionally, the paper includes information on IAAs for both rounds of annotations, demonstrating the rigor and reliability of the dataset.\n3. The task of evaluating the level of substantiation in peer reviews is highly relevant to the NLP community.\n4. While the paper focuses on the NLP domain, the concept of claim-evidence pair extraction and substantiation evaluation can be applicable to other research fields beyond NLP.", "reasons_to_reject": "1. The evaluation is done on a quite limited and domain-specific data collection. This is acceptable for the task itself but the third claim on the contribution must be better clarified (e.g., \"We provide meaningful insights into the current level of substantiation in scientific peer reviews.\" is only valid for a specific domain and limited data points)\n\nNo other major issues were identified in the paper, just some minor remarks:\n1. The sentiment classifier used in the paper is based on tweets, this could potentially bias the baseline.\n2. The models used in the experiments are identified by (I suppose) HF hub tags, this should be explicitly mentioned.", "questions_to_authors": "A. Could you provide any justification for using a sentiment classifier based on tweets for the baseline?\n\nB. How do you plan to address the potentially limited generalizability of your approach and findings? Have you considered any strategies to adapt or validate your approach on reviews from other research domains to ensure broader applicability?", "paper_topic_and_main_contributions": "This paper addresses the problem of problematic reviews in NLP (not AI as suggested) conferences. It proposes an automatic quality control measure focusing on the aspect of \"substantiation\" (the level of support and evidence for the claims made in a paper review). The paper formulates the problem as claim-evidence pair extraction in scientific peer reviews and introduces a new, very curated dataset, which should be the first annotated dataset for this task. The dataset is not large in the \"2023\" sense, with 550 reviews from NLP conferences, annotated by domain experts, however, this is a good starting point for a dataset requiring such a level of specialized knowledge to be annotated.\n\nContributions:\n- The paper introduces the SubstanReview dataset, which is a valuable resource for studying the quality of peer reviews in NLP conferences.\n- The paper develops a system trained on the new dataset. It can automatically analyze the level of substantiation in reviews, enabling automatic quality control measures for problematic reviews.\n- Problematic peer reviews can have serious consequences for the scientific community, leading to incorrect decisions on paper acceptance or rejection. By focusing on substantiation and providing an automated \"solution\" for the issue.\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CRwPXPUfKc", "reviewer": "Reviewer_3y4q", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our paper is well written, that it addresses an important topic and opens up opportunities for novel research in this underdeveloped area. However, there seems to be misunderstandings concerning key elements of our paper which we would like to clarify.\n\n1. While the size of our dataset is limited, the results in our paper demonstrate our dataset to be sufficient for fine-tuning BERT-based models. Moreover, in future work, our dataset could also be used for instruction tuning and in-context learning, where only a small amount of high-quality human curated data is required. Therefore, there are clear use-cases for our dataset of limited size. We furthermore want to add that recently published annotated datasets in this domain contain fewer samples than our SubstanReview dataset [1]. Statistics of the reviews from our annotations are shown in Table 2. For other details of the reviews, please refer to the original NLPeer paper [2], we cannot include all of them due to space limitations. Unfortunately, no information regarding the reviewers is publicly released.\n\n2. The baseline is not fine-tuned on our dataset and thus expected to achieve lower performance than the fine-tuned models. We introduce the baseline in Lines 536 –549, it is a heuristic approach based on an open-source sentiment classifier and BERTScore. In Table 3, BERT, RoBERTa, SciBERT and SpanBERT are fine-tuned on our dataset while the baseline is not. The fact that models fine-tuned on SubstanReview significantly outperform the  baseline prove that our dataset is efficient for the claim-evidence pair extraction task. We will further emphasize and clarify this point in the final version of our paper.\n\n3. We have conducted a systematic study to validate our intuition, please refer to Section 4.4 SubstanScore. There, we calculate the correlation between human annotated substantiation scores and SubstanScore derived from our intuition, resulting in a significant Spearman’s ρ = 0.7568. \n\n[1] Fromm, Michael, et al. \"Argument mining driven analysis of peer-reviews.\" AAAI 2021.\n\n[2] Dycke, Nils, Ilia Kuznetsov, and Iryna Gurevych. \"NLPEER: A Unified Resource for the Computational Study of Peer Review.\" ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem is very relevant in the scientific community as more and more papers are getting published.\n- Dataset opens doors for new research in this often ignored area\n- Paper is very well written and is a good read especially to understand how review quality can be measured", "reasons_to_reject": "- Even though the new dataset opens up a lot of opportunities for research in this area, 550 reviews in the area of NLP are very limited, especially considering that these international conferences and reviewers from different regions have different styles. Not many details have been provided on the characteristics of reviews and reviewers. This part is also missing in the evaluation.\n- Baseline achieves lower performance. It is not clear if that is due to the limited size of the dataset or the nature of the problem\n- It seems intuitive to measure the substantiation by counting the number of claims with evidence, there is no systematic proof or any study that proves that this property is always true.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper solves the problem of enforcing review quality by measuring the substantiveness of the claims made in reviews. To further study the problem, a new dataset is created that provides claim-evidence pairs for 550 reviews. This dataset is used to set up a baseline for this task using different models. ", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper under review addresses a key issue in the scientific community - improving the quality of peer reviews. It focuses on the aspect of substantiation, which is defined as the degree to which claims in a review are supported by evidence. The authors frame the issue as a claim-evidence extraction task and introduce a new dataset (SubstanReview) for this purpose. This dataset, although small and domain-specific (550 reviews from NLP conferences), is the first of its kind, and the reviewers acknowledge its potential value to the NLP community. The creation of this dataset and the development of a system to automatically analyze the level of substantiation in reviews are the main contributions of the paper. The reviewers generally agree that these are significant and valuable contributions. They also commend the clarity of the paper and its potential applicability beyond the task-specific domain.

However, the reviewers also raise some concerns. Firstly, the reviewer's main concern was on dataset's limitedness being applied to NLP models. Secondly, the evaluation of the system is seen as limited due to the domain-specific nature of the data collection. Questions were also raised about the generalizability of the findings and the potential bias in the baseline sentiment classifier, which is based on tweets. Thirdly, some of the claims made in the paper were seen as lacking sufficient support. For instance, the substantiation score is not discussed extensively in the paper, and the relationship between the number of claims with evidence and the level of substantiation is assumed without systematic proof.

Overall, while the paper makes a significant contribution to the field and could potentially impact the way peer reviews are conducted, there are several important issues that need to be addressed to enhance the robustness and generalizability of the findings. The reviewers' scores reflect these mixed feelings, with some expressing strong support and others being more ambivalent or even critical. Regardless of the concerns discussed by the reviewers, this paper has substantial merits that can affect the broader communities of NLP.
Decision: Accept-Findings