When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews
Authors: Sandeep Kumar
Tirthankar Ghosal
Asif Ekbal
Venue: EMNLP 2023 Main (2023)
Keywords: Contradiction Detection
Peer Reviews
NLP
TL;DR: We introduce "ContraSciView", a dataset for identifying contradictions in peer reviews, and propose a baseline model for detecting these contradictions, marking the first automated effort to spot disagreements among reviewers.

Abstract: To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, $\textit{identify points of agreement and disagreement}$ and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce $\textit{ContraSciView}$, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.

When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews
Sandeep Kumari, Tirthankar Ghosali, Asif Ekbalt iIndian Institute of Technology Patna, India
National Center for Computational Sciences, Oak Ridge National Laboratory, USA ((sandeep_21211g 2121c2222222 asi aswi)siiti.e..... t@ornl gov
Abstract
Fleradige sing vitt anabaa proderg Dunn For Trear commooso eigh oxperiecet proviee support fidd
To this date, the efficacy of the scientific pub- lishing enterprise fundamentally rests on the strength of the peer review process. The jour- nal editor or the conference chair primarily re- lies the expert reviewers' assessment, iden- rify points of agreement and disagreement and try to reach a consensus to make a fair and in- formed decision whether to accept or reject paper. However, with the escalating number of submissions requiring review, especially in top- tier Artificial Intelligence (AT) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to miti- gate reviewer disagreements. Here in this work, we introduce a novel task of automatically iden- tifying contradictions among reviewers given article. To this end, we introduce Con- traSciView, a comprehensive review-pair con- tradiction dataset around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review- based ICLR and NeurIPS conferences. We fur- ther propose a baseline model that detects con- tradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations'.
Barkersnd
tocoee raher chocca sonting toayy and In saart to dis- aviercc Fwentii dee rernodeew sccepiing x.ppe...
Figure I: An example of contradiction among reviewers.
all been subjects of concern within the scholarly community. The rapid increase in research article submissions across different venues has caused the peer review system to undergo a huge amount of stress (Kelly et al., 2014; Gropp et al., 2017). The role of an editor in scholarly publishing is crucial (Hames, 2012). Typically, editors chairs have to manage multitude of responsibilities. These include finding expert reviewers, assigning review tasks, mediating disagreements among reviewers, ensuring reviews are received on time, stepping in when reviewers are not responsive, making in- formed decisions, and maintaining communication with authors, among other duties. Simultancously, they must ensure the validity and quality of the re- views to make informed decision. However, the complexity of scholarly discourse and inherent sub- jectivity within research interpretation sometimes leads to conflicting views among peer reviewers (Bormaann and Daniel, 2009; Borcherds and Ed- itor, 2017). For instance, in Figure 1, Reviewer regards the evidence as both strong and suffi- cient, reinforeing the paper's theoretical soundness. Conversely, Reviewer 2 remains skeptical of this evidence, underscoring their differing perspectives on the soundness of the paper.
1 Introduction
Despite being the widely accepted standard for val- idating scholarly research, the peer-review process has faced substantial criticism. Its perceived lack of transparency (Wicherts, 2016; Parker al., 2018).
sometimes being biased (Stelmakh et al., 2021,
2019), arbitrariness (Brezis and Birukou, 2020),
Feedback between authors and reviewers can help improve the peer review system (Rogers and Augenstein, 2020b). However, reviewer disagree- ment can create confusion for authors and editors as they try address reviewer feedback. There are various guidelines or suggestions that editors consider in order to resolve the conflicting reviews (Borcherds and Editor, 2017). Given this volume, it
inconsistency (Shah et al., 2018; Langford and Guz- dial, 2015), being regarded as a poorly defined task (Rogers and Augenstein, 2020a) and failure to rec- ognize influential papers (Freyne et al., 2010) have
'https://ee http://gttuuuccmmmmeeeeeeeeeeeee 20m/ Contradictions ienetiiionnn-eeeeee ac. iniiiiniiiii ai-nlp nlp-mi ml mamemml/rreeeeeee /tt//ouuuucss... html #ContraSciiiew
and https: /www///ww.t. ii

---

is challenging for editors identify contradictions manually. Our current investigation can assist in pinpointing these discrepancies and help editors make informed decisions.
Venues
Papers
#Reviews
# PParrs
ICLR
5,096
14,711
14,383
NeurlIPS
3,486
11,114
14,114
Total
8,582
25,825
28,497
It is to be noted that this Al-based system aims aid editors in identifying potential contradictions in reviewer comments. While it provides valuable insights, not infallible. Editors should use as a supplementary tool, understanding that not all contradictions may be captured and manual review remains crucial. They should make decision with careful analysis beyond the system's recommenda- tions.
Table 1: Dataset statistics
text pairs into entailment, neutral, contradiction (Chen al., 2017; Gong et al., 2018). Well-known transformer-
models, such as BERT (Devlin et al., 2019) and its variants leading to state-of-the- art performance.
As far as we know, contradiction detection peer review has never been studied. Contradiction detection in peer reviews is complex and requires domain knowledge of the subject. It is not straight- forward to detect contradictions in peer reviews be- cause reviewers often have different writing styles and approaches to commenting, We believe that our work can significantly contribute the peer review process.
Our contributions are three-fold: We intro- duce a novel task: identifying contradictions/dis- agreement within peer reviews. 2) To address the task, we create a novel labeled dataset of around 8.5k papers and 25k reviews. 3) We establish baseline method as reference point for further research this topic.
2 Related Work
Dataset
Artificial Intelligence (AI) has been applied in re- cent years to the realm of scientific peer review with the goal of enhancing its efficacy and quality (Checco et al., 2021; Ghosal et al., 2022). These applications span a diverse range of tasks, includ- ing decision prediction (Kumar et al., 2022; Ghosal et al., 2019), rating prediction (Li et al., 2020; Kang al., 20), sentiment analysis (Kumar al., 2021; Chakraborty al., 2020; Kang et al., 2018b), argument mining (Hua al., 2019; Cheng al., 2020), and review summarization (Xiong, 2013). this work, we aim to broaden the scope of AI's usefulness by assisting the editor in determining the disagreement between reviewers. Contradiction detection is not new. Alamri and Stevenson (2015) use linguistic features and support vector machines (SVMs) detect contradictions in scientific claims, while Badache al. (2018h) employ review po- larity to predict contradiction intensity in reviews. et al. (2018) combine sentiment analysis with contradiction detection for Amazon reviews. Mean- while, Lendvai and Reichel (2016) employ textual similarity features to classify contradictions in ru- mors, and Li et al. (2017) contradiction-speiiin word embeddings for sentence pairs. Tan et al. (2019) propose dual attention-based gated recur- rent unit for conflict detection. Recent advances in Natural Language Inference (NLI) have brought forth a category of textual entailment, categorizing
3.1 Dataset Collection
We utilize a subset of 8,582 out of 8,877 papers from the extensive ASAP-Reviee dataset (Yuan et al., 2021). The dataset comprises reviews from the ICLR (2017-2020) and NeurIPS (2016-2019 conferences. Each review is labeled with eight as- pect categories: (Motivation, Clarity, Soundness, Substance, Originality, Meaningful Comparison, Replicability, and Summary) along with their senti- ment (Positive/Negative) except Summary.
3.2 Dataset Pre-processing
We define review is as a collection of com- ments/sentences written by one reviewer. Formally, we can represent it as list:
R {cmt1 emtg....)
A review pair takes two such lists, one from Reviewerl and one from Reviewer2. It can be represented as:
RP = (R1, R2)
Lastly, review pair comment selects one com- ment from each reviewer and forms pair. It is of such pairs:
RPC = {(cmt from R1, cmt from R2)....}

---

To make it easier annotate, we first create pairs of reviews of papers. Suppose, if there are n number of reviews in a paper then we create (2) pairs resulting in a total of around 28k pairs. De- tailed statistics regarding this dataset can be found in Table 1. We follow the contradiction definition of Badache et al. (2)18a). According to this deñi nition, contradiction exists between two review pairs when any review pair comments, denoted as raj and rag, contain common aspect category but convey opposite sentiments. Therefore, we categorize those review pairs as no contradiction if none of their comments share the same aspect with opposing sentiments. This labeling process resulted in 17,095 pairs of reviews. For the remain- ing review pairs, we compile a list of review pair comments that share the same aspect but express opposing sentiments, and we designate these for human annotation. Finally, we annotate a total of 50,303 pair of review pair comments.
review pairs from this more extensive train our annotators, ensuring both classes are equally represented. Upon completion of this round of annotation, we reviewed and corrected any misin- terpretations with the annotators, further refining their training and enhancing the clarity of the an- notation guidelines. To evaluate the effectiveness of the initial training round, we compiled another 80 review pairs from both classes drawn from the remaining review pairs. From the second round on wards, most annotators demonstrated increased proficiency, accurately annotating at least 70% of the contradictory cases.
4.3 Annotation Process
We regularly monitored the annotated data, placing emphasis on identifying and rectifying inconsis- tencies and cases of confusion. We also imple- mented an iterative feedback system that continu- ously aimed to refine and improve the annotation process. In cases of conflict or confusion, we con- sulted experts to make the final decision. Following the annotation phase, we obtained an average inter- annotator agreement score of 0.62 using Cohen's 's kappa (Cohen, 1960), indicating a substantial con- sensus among the annotators.
4 Annotating for Contradiction
4.1 Annotation Guidelines
We follow the contradiction definition by De Marn effe et al. (2008) for our annotation process where they define contradiction as: "ask yourself the fol- lowing question: If I were shown two contempo- raneous documents, one containing each of these passages, would I regard it as very unlikely that both passages could be true at the same time? If so, the two contradict each other". We offer multi- ple examples from different aspect categories that may contain conflicting reviews assist and direct the annotators. Figure illustrates the flowchart that represents our annotation procedure. For the detailed annotation guidelines, please refer to Ap- pendix A.
4,4 Annotator's Pay
We compensated each annotator based on stan- dard salaries in India, calculated by the hours they worked. The appointment and salaries are gov- erned by the standard practices of our university. We chose not to pay per review pair because the time needed to understand reviews varies due to their complexity, technical terms, and the annota- tor's s familiarity with the topic. Some reviews are also extensive, requiring more time comprehend. Hence, basing pay on review pairs could have com- promised annotation quality. To ensure accuracy and prevent fatigue, we set daily limit of 6 hours for annotators.
4.2 Annotator Training
Given the complexity of the reviews and their fre- quent use of technical terminology, we had six doc- toral students, each with four years of experience in scientific research publishing. To facilitate their training, two experts with more than ten years of experience in scientific publishing annotated 1,500 review pairs from selection of random papers, following our guidelines. Our experts convened to discuss and reconcile any discrepancies in their an- notations. The initial dataset comprises 227 pairs with contradictions and 1273 pairs without con- tradiction comments. We randomly selected 100
4.5 Final Dataset
Figure 2 displays the annotation statistics. We ob- served that the majority of disagreements among reviewers pertain to the paper's clarity. Such dis- agrecments could stem from differences in review- ers' expertise, domain knowledge reviewer unfa- miliar with the domain might find it hard grasp the content), language proficiency (some reviewers might struggle with English, while others are flu- ent), or interest level (a disinterested reader might

---

14000
task model (Kumar et al., 2021). A review pair com- ment can have different sentiments corresponding to multiple aspect categories. So, we utilize Multi- Instance Multi-Label Learning Network (MIM- LLN) (Li et al., 2020) which uniquely identifies sentences as bags and words as instances. It func- tions multi-task model, which performs both Aspect Category Detection (ACD) and Aspect Cat- egory Sentiment Analysis (ACSA). Given a tence and its aspect categories, predicts sentiment by identifying key instances, and aggregating these to obtain the sentence's sentiment towards each category. We discuss MIMLLN in more detail Appendix B. We trained MIMLLN on the human- annotated ASAP- dataset for our task.
12000
8030
4030
2030
dailiyy
meaningfol
motivation
orginality
epplicablitt
Aoundness
substance
Figure2 2: Comparison of Contradiction by Aspect; y axis: count of review comment, aspect category, c: contradiction, n: non-contradiction
Sentiment Disparity Aspect Pairs
Yes
Aspect Sentiment
Reviewer Disagreement Classifier We use tech- niques from Natural Language Inference (NLI) sen- tence pair classification to identify reviewer dis- agreement, particularly contradictions from Sen- lence Dependency Pairs (SDPs). Unlike traditional NLI tasks that provide three-category output of "entailment", "contradiction", and "neutral", we have adjusted the model to two-category output system: "contradiction" and "non-contradiction" The latter category combines "entailment" and "neutral" labels, as our primary focus is on con- tradiction detection.
(1)
ASOP?
Model
Sentiment
Reviewer Disagreement Classifier
(2)
Disparity
Contradictions
Aspect
Pairs
Figure 3: Flowchart of our proposed baseline. Here, RI and R2 represent the two reviews. baseline consists of two steps: (1) extracting the SDAPs and (2) deter- mining whether these SDAPs are contradictions ASOP stands for "Is Aspect same and sentiment opposite?"
find the paper difficult engage with). Discrepan- cies regarding replicability and meaningful compar- ison are notably fewer, likely because these topics are less frequently commented on.
Results and Analysis
We discuss the implementation details in Appendix C. Table 2 reports the performance of the models when trained on our dataset. The results of the Reviewer Disagreement Classifier, which is the fi- nal step in our proposed approach. We report maero average R, and scores as the rarity of the con- tradiction class is particular interest for this task (Gowda et al., 2021). RoBERTa large outperforms XLNet large by 0.7 points, RoBERTa base by 4.1 points, XLNet by 4.5 points, and DistilBERT by 7.5 points with respect to Fl score.
5 Baseline System Description
We describe the flow of our proposed baseline setup through the flowchart in Figure 3. The initial in- put involves pair of reviews for a given paper, which are subjected to the Aspect Sentiment Model. This model classifies the aspect and sentiment of each review comment/sentence within the reviews. Subsequently, identify pairs of comments that, while sharing the same aspects, exhibit differing sentiments; we term it as Sentiment Disparity As- pect Pair (SDAP). As a final step, the SDAPs are then passed to the Contradiction Classifier in or- der to classify whether these pairs of review com- ments are contradictory or not. We describe the Aspect Sentiment Model and Contradiction Detec- tion Model in details follows:
In order to analyze how models perform when trained on natural language inference datasets, we trained the models on the ANLI+ALL dataset and evaluated them on our test set. It was found that the models trained by combining datasets, i.e., SNLI, MNLI, FEVER, and ANLI A1+A2+A3, perform the best (Nie et al., 2020). We obtain the high- Fl score of 71.14 F1 with RoBERTa Large. The results show that all the models trained on our dataset outperform those trained on the existing datasets. This large increase in performance can
Aspect Sentiment Model: Aspect and senti- ment in peer review have been studied as a multi-

---

Model
P
R
F1
Acc
Finetuned on (ANLI +ALL)
DistilBERT (Sanh et al., 2019)
59.55
67.30
63.13
77.34
XLNet base(Yang et al., 2019)
63.16
75.47
68.75
75.72
RoBERTa base (Liu et al., 2019)
63.15
75.63
68.90
78.53
XL.et Large (Yang et al., 2019)
63.26
76.11
69.22
79.20
RoBERTa large (Liu et al., 2019)
65.32
78.03
71.14
81.34
Trained on our dataset
DistiBBERT
69.50
70.69
70.07
87.70
XLNet base
71.13
75.68
73.05
88.07
RoBERTa base
74.49
72.43
73,40
89.83
XLNet large
74.41
80.00
76.76
89.64
RoBERTa large
79.17
76.17
77.55
91.50
Table 2: Results of Contradiction Classifier; P: Precision, R: Recall, Fl: FI-Score, Acc: Accuracy
be attributed to the differences in reviews and the content of the existing datasets. Scientific peer re- view comments are written differently from typical human-written English sentences and Wikipedia entries. They contain a more technical style writ- ing, which is challenging for models to parse when trained on current datasets. Our findings, therefore, highlight the pressing need for innovative datasets in this specific domain. We discuss the results of our proposed intermediate aspect sentiment model in Section D.
found that Google BARD registered higher num- ber of false positives compared to CHATGPT.
Conclusion and Future Work
In this work, we introduce a novel task au- tomatically identify contradictions in reviewers' comments. We develop ContraSciView, a review- pair contradiction dataset. We designed baseline model that combines the MIMLLN framework and NLI model to detect contradictory review pair com- ments. We found that RoBERTa large performed best for this task with Fl score of 71.14.
Next, we evaluate the performance of RoBERTa Large across the entire process (in evaluation mode). We obtain an accuracy of 88.60% from the Aspect Sentiment Classifier (in determining whether pair of reviews has any SDAP or not) and 74.25 Fl score for the Reviewer Disagrecment Classifier. We compare achieved by the zero-shot Large Language Model, ChatGPT. On the test set, ChatGPT scored an F1 64.67 which is 9 points lower than the bascline model, likely due to its lack of explicit training for this specific downstream task. We discuss the prompts and outputs in details in the Appendix E. We also discuss where our proposed baseline fails (Error Analysis) in the Appendix F. We also utilized the BARD API for evaluation. We provided identical prompts to both Google BARD and CHATGPT ensure fair comparison. We compared our findings with those achieved by Google BARD. BARD scored an Fl of 61.35 on the test set, 12 points lower than the baseline model. This is likely due to its BARD's requirement for more specialized training for this specific task. We
Our proposed framework doesn't ''t consider the full review context when predicting contradictions. In the future, we will investigate system that takes into account the entire review context while detect- ing contradictory review pair comments. Addition- ally. we plan to expand our dataset to include other domains and explore the significance of Natural Language Inference for the given task. We also aim categorize contradictions based on their severity: high, medium, or low.
findings with those
Limitations
Our study mainly focuses on identifying "explicit" contradictions. Explicit Contradictions: These are clear, direct, and unmistakable contradictions that can be easily identified. For example: The author claims in the introduction that "X has been proven to be beneficial," but in the discussion sec- tion, they state that "X has not been shown to have any benefit." One reviewer says, "Figure 2 clearly shows the relationship between A and B," while another reviewer comments, "Figure 2 does not show any clear relationship between A and B."
Phttps: http:://ggghhuuccoomeeeeeeee com/
API

---

We do not delve into "implicit" contradictions. which can be hard detect and can be subjective, making them topic of debate, Implicit Contra- dictions:
reviewers will help editors identify and initiate discussion to resolve the confusion between the reviewers and to make an informed decision. Since reviews are frequently extensive and intricate, it is challenging for editors to scrutinise every comment and address conflicts. This system aims to aid them in spotting such contradictions. For instance, if one comment reads, "The paper does not have any new findings," and another reviewer mentions, "The paper is somewhat novel," the editor might not im- mediately perceive this as contradictory. However, understanding the context and intent behind these comments is vital. they represent a contradiction, editors can address it using the standard guidelines. The system does not detect all contradictions. If the system fails to identify contradiction, it does not automatically mean none exists. Given its nature general AI system, there is a possibility of it pre- senting false negatives. Editors relying solely on this system could impact the review process. Any contradictions spotted outside the system's recom- mendations should also be addressed according to the guidelines. As an AI based model this is prone to errors, the editor/chair are advised to utilize this tool only for assistance and verify the contradiction and analyze carefully before making decision.
These are more subtle and may require deeper understanding or closer examination to iden- tify. It may require the annotators to read the paper and also learn many related works or things to anno- tate. They are not directly stated but can be inferred from the context how information is presented. For example: Review 1: "the method lacks algo- rithmic novelty and the exposition of the method severely inhibits the reader from understanding the proposed idea " Review 2: "the work presented is novel but there are some notable omissions- there are no specific numbers presented to back up the improvement claims graphs are presented but not specific numeric results there is limited discus- sion of the computational cost of the framework presented there is no comparison to baseline in which the additional learning cycles used for learn- ing the embedding are used for training the student model For instance, Reviewl mentions that the method lacks algorithmic novelty, and Review2 acknowledges the work as novel but points out some notable omissions. This difference in percep- tion is not direct contradiction, one reviewer finds the method lacking in novelty, while the other recognizes it as novel but with some omissions. Additionally, we do not incorporate information from the papers being reviewed, as they cover a wide range of topics and would require many ex- perts from various domains. However, we acknowl- edge that finding a method to uncover these implicit contradictions in reviews an intriguing opportu- nity for future research.
A Acknowledgement
Sandeep Kumar acknowledges the Prime Minister Research Fellowship (PMRF) program of the Govt of India for its support.
References
Abdulaziz Alamri and Mark Stevenson. 2015. Au- tomatic identification of potentially contradictory claims to support systematic reviews. In 2015 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2015, Washington, DC, USA, November 9-12, 2015, pages 930-937. IEEE Com- puter Society.
Ethics Statement
We have utilized the open source peer review dataset for our work. The system sometimes gen- erate incorrect contradictions or overlook certain contradictions. Like other general Al models, this model is not 100% accurate. Editors or Chairs might rely on more than just the contradictions predicted by the model to make decisions. It is important to emphasise that the primary purpose of this system is to assist editors by highlighting potential contradictions between reviewers; this system only for editors' internal usage, not for authors or reviewers. Especially given their of- ten busy schedules and the myriad of decisions they must make. The contradictions among the
Ismail Badache, Sébastien Fournier, and Adrian- Gabriel Chifu. 2018a. Contradiction in reviews: Is strong or low? In Proceedings of the First International Workshop on Analysis of Broad Dynamic Topics over Social Media (BraDyn 2018) co-located with the 40th European Conference on Information Retrieval (ECIR 2018), Grenoble, France, March 26, 2018 vol- ume 2078 CEUR Workshop Proceedings, pages 11-22. CEUR-WS.org
Ismail Badache, Sébastien Fournier, and Adrian-Gabriel Chifu. 2018b. Predicting contradiction intensity: Low, strong or very strong? In The 41st International ACM SIGIR Conference on Research & Development

---

Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1125-11 1128. ACM.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, NAACL-IILTT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages
Iz Beltagy, Arman Cohan, and Kyle Lo. 2019. Scibert: Pretrained contextualized embeddings for scientific text. CoRR, Htss/993.106677
Anne Borcherds and Managing Editor. 2017. How deal with conflicting reviewer comments.
Association for Computational
Linguistics.
Lutz. Bornmann and Hans-Dieter Daniel. 2009. The luck of the referee draw: the effect of exchanging reviews. Learn. Publ. 22(2): 17-125.
Jill Freyne, Lorean Coyle, Barry Smyth, and Padraig Cunningham. 2010. Relative status of journal and conference publications in computer science. Com- mun. ACM, 53(1 124-132.
Elise S.
Brezis and Aliaksandr Birukou. 2020. Arbi- trariness in the peer review process. Scientometrics. 123(1):393-411
Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. PLOS ONE, 17(1):1-29.
Souvie Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analy- sis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 207-216.
Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Push- pak Bhattacharyya. 2019. DeepSentiPeer: Harness- ing sentiment in review texts to recommend peer review decisions. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 1 1120-1130, Florence, Italy. Asso- ciation for Computational Linguistics.
Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. Ai-assisted peer review. Humanities and Social Sci- ences Communications, 8.
Yichen Gong, Heng Luo, and Jian Zhang. 2018. Nat- ural language inference over interaction space. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. 55th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2017, Vancouver, Canada, July 30- August 4, Volume Long Papers, pages 1657-1668. Association for Computational Linguis- tics.
Proceedings of the
Thamme Gowda, Weiqiu You, Constantine Lignos, and Jonathan May. 2021. Macro-average: Rare types are important too. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1138-1157. Association for Com- putational Linguistics.
Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 7000-7011, Online. Association for Computa- tional Linguistics.
Robert E. Gropp. Scott Glisson, Stephen Gallo, and Lisa Thompson. 2017. Peer Review: A System under Stress. BioScience, 6775):407-410.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: document-leww representation learning using citation- informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguisties, ACL 2020, Online, July 5-10, 2020, pages 2270-2200 -2822. Association for Computational Linguistics.
Irene Hames. 2012. peer review in a rapidly evolving publishing landscape. In Robert Campbell, Edd Pentz, and Ian Borthwick, editors, Academic and Profes- sional Publishing, pages 15-52, Chandos Publishing.
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2131-2137, Minncapolis, Minnesota. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20:37 46.
Marie-Catherine De Marneffe, Anna N Rafferty, and Christopher D Manning. 2008 Finding contradic- tions text. In Proceedings of acl-08: Hit, pages 1039-10477
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard

---

Hovy, and Roy Schwartz. 2018a. dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 1647-1661, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLE: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages Association for Computational Linguisties.
Timothy Parker, Simon Griffith, Judith Bronstein, Fiona Fidler, Susan Foster, Hannah Fraser, Wolfgang Forstmeier, Jessica Gurevitch, Julia Koricheva, Ralf' Seppelt, Morgan Tingley, and Shinichi Nakagawa. 2018. Empowering peer reviewers with checklist to improve transparency. Nature Ecology Evolution, 2:929-935.
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Ed- uard Hovy, and Roy Schwartz. 2018b. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint arviv: 1804.09635.
Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. 2014. Peer review in scientific publications: Benefits, critiques, a survival guide. E/IFCC, 25:227 43.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NewrlPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Sandeep Kumar, Hardik Arora, Tirthankar Ghosal, and Asif Ekbal. 2022. Deepaspeer: towards an aspect- level sentiment controllable framework for decision prediction from academic peer reviews, In Proceed- ings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. pages 1-11.
Sandeep Kumar, Tirthankar Ghosal, Prabhat Kumar Bharti, and Asif Ekbal. 2021. Sharing is caring! joint multitask learning helps aspect-category extraction and sentiment detection in scientific peer reviews. In ACM/IBEE Joint Conference Digital Libraries, JCDL 2021, Champaign, IL, USA, September 27-30, 2021, pages 270-273. IEEE.
Anna Rogers and Isabelle Augenstein. 2020a. What can we do to improve peer review in NLP? In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1256-1262, Online, Association for Computational Linguistics.
John Langford and Mark Guzdial. 2015. The arbitrari- ness of reviews, and advice for school administrators. Commun. ACM, 58(4): 12-13.
Anna Rogers and Isabelle Augenstein. 2020b. What can we do to improve peer review in nip? In Find- ings the Association for Computational Linguistics: EMNLP 2020, Online Event. 16-20 November 2020, volume EMNLP 2020 of Findings of ACL. pages 1256-1262 Association for Computational Linguis- tics.
Piroska Lendvai and Uwe D. Reichel. 2016. Con- tradiction detection for rumorous claims. CoRR, abs/1611.00258.
Chuqin Li, Xi Niu, Ahmad Al-Doulat, and Noseong Park. 2018. A computational approach to finding con- tradictions in user opinionated text. In IEEE/ACM 2018 International Conference on Advances in So- cial Networks Analysis and Mining, ASONAM 2018, Barcelona, Spain, August 28-31, 2018, pages 351- 356. IEEE Computer Society.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CORR. abs/1910.01108.
Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike von Luxburg. 2018. De- sign and analysis of the NIPS 2016 review process. J. Mach. Learn Res., 19:49:1-49:33
Luyang Li, Bing Qin, and Ting Liu. 2017. Contra- diction detection with contradiction c word embedding. Algorithms, 10(2):59.
Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. 2019. On testing for biases in peer review. In Advances in Neural Information Processing Systems 32: An- nual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5287-5297.
Yuncong Li, Cunxiang Yin, Sheng- -haag-huaaue Zhong, and Xu Pan. 2020. Multi-instance multi-label learn- ing networks for aspect-category sentiment analysis. arXiv preprint arXiv:2010.02656.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1 11692.
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, and Hal Daumé III. 2021. Prior and prejudice: The novice reviewers' bias against resubmissions in con- ference peer review. Proc. ACM Hum. Comput. In- teract,, 5(CSSCCCWWC))311.5.511

---

Yes
the viewpoints expressed by the reviewers. Take note of their opinions, arguments, and any specific aspects they highlight.
Read the Title and Abstract of the pager carefully,
pair review comments contradicting CNT provided the espect category?
Reed the Pair of Review text carefully
Step 3: Based on the reviewer comments, their aspect- sentiment analysis, and the pair of reviews, you should categorize them accordingly. If the reviews contradict each other, indicating oppos- ing viewpoints or conflicting statements, mark it as a contradiction (C). there is no contradiction or the reviews are unrelated, mark them as non- contradiction (N). In cases where you find it diffi cult to determine if there is a contradiction or not, mark it as CON (confused).
No
Figure 4: A step-by-step flowchart used by anno- tators to annotate any pair of review contradiction; CesCrrttaai,iion, No:Not Contradiction, CNT:Cannot Decide
Xingwei Tan, Yi Cai, and Changxi Zhu. 2019. Recog- nizing conflict opinions in aspect-level sentiment clas- sification with dual attention networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Join! Conference on Natural Language Pro- cessing, EMNLP-IJCNLA 2019, Hong Kong, China, November 3-7, 2019, pages 1424-3429, Association for Computational Linguistics.
To decide if the reviews are contradictory, ask yourself the following question: "If I were shown two contemporaneous documents one containing each of these passages, would regard as very un- likely that both passages could be true at the same time? If so, the two contradict each other." You should be able to state a clear basis for a contradic- tion. For example, the following are contradictions: R1. The motivation of using the method for a very small improvement is not convincing.
J.M. Wicherts. 2016. Peer review quality and trans- parency of the peer-review process in open access and subscription journals. PLOS ONE, 11(1).
Wenting Xiong. 2013. Helpfulness-gun guided review sum- marization. In Proceedings of the 2013 NAACL HLT Student Research Workshop, pages 77-83, Atlanta, Georgia. Association for Computational Linguistics.
R2. Overall feedback: found the paper to be well motivated and the proposed approach to be interesting.
How to make use of the aspect category while annotation? Given the assigned aspect calegory, you should utilize the given category to accurately compare these comments. This will help keep your attention strictly on the relevant aspect and refrain from de- viating to other topics mentioned in the review text. For example².
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quac V. Lee 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural In- formation Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NewrlPS 2019, December& 8- 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021. Can we automate scientific reviewing? CoRR, abs/2102.00176.
R1: However, the results are a visible improve- ment over JPEG 2000 and I don't 't know of any other learned encoding which has been shown to achieve this level of performance.
A Annotation Guidelines
R2: ...the comparison to JPEG2000 is unfortu- nately not that interesting since that codec does not have widespread usage and likely never will.
Recognizing Contradiction in a pair of reviews in- volves analyzing two pieces of text written by two different reviewers about the same paper. The fol- lowing steps can be followed:
In Review 1, the reviewer values the comparison made with JPEG 2000, considering it suitable benchmark and a positive aspect of the paper. Con- versely, the reviewer in Review 2 argues that the comparison with JPEG 2000 is not compelling due to its limited use and suggests a comparison with WebP would be more relevant. These statements present opposing viewpoints the relevance of comparing with JPEG 2000 in the paper. While
Step 1: Begin by reading the Title and Abstract of the paper to gain an understanding of its subject matter. It is important to read these sections multi- plc times to grasp the paper's main points, such as its motivation, contributions, and other relevant as- pects. If necessary, refer the paper itself read related material to enhance your understanding. Step 2: Proceed to read and comprehend the reviews of the paper, focusing on understanding
'Paper id of the review: ICLR_ 2017_217_22

---

there are many topics discussed in Review 1 and Review 2, crucial to concentrate on the re- marks concerning the comparison between JPEG and other potential benchmarks when assessing these reviews.
ACD segment comprises four modules: an embed- ding layer, an LSTM layer, an attention layer, and an aspect category prediction layer. Similarly, the ACSA segment includes four components: an em- bedding layer, a multi-layer Bi-LSTM, a word sen- timent prediction layer, and an aspect category sen- timent prediction layer. In the ACD task, all aspect categories utilize the same embedding and LSTM layers, but they have distinct attention and aspect category prediction layers. For the ACSA task, all aspect categories share the embedding layer, the multi-layer Bi-LSTM, and the word sentiment prediction layer, yet they each have unique aspect category sentiment prediction layers.
When comparing reviews, keep in mind that one might discuss the subject matter in broad terms, while the other may focus on specific elements. Consider this example:
RI: The work is original.
R2: The extension to the partially observable setting is interesting as the proposed form finds common denominator to multiple estimators, but its underlying idea is not novel.
In this case, Reviewer 1 comments on the overall originality of the paper, declaring it as novel. On the other hand, Reviewer 2 critiques a specific part of the paper- the extension to the partially observ- able setting-and declares this particular aspect as not novel. Even though Reviewer 2's comment is more specific, it contradicts the general assertion made by Reviewer about the paper's novelty.
C Implementation Details
We implemented our system using Py ToTch (Paszke et al., 2019), deep learning framework, and uti- lized pre-trained transformer models from Hugging Face The dataset was randomly split into three parts: 80% for training, 10% for validation, and 10% for testing.
It is important to identify these types of exam- ples as contradictions, even if they may seem to operate at different levels of specificity. The broad comment about the paper's novelty in Review is contradicted by the specific critique Review You may find more detailed annotation guide- lines in our shared repository.
For the Aspect Sentiment Model, we conducted experiments with different network configurations during the validation phase. ASAA-Review dataset contains predictions labelled by an aspect tag- ger on a human-annotated label. We used the 1,000 human- aanoantd reviews, maintaining the same random split, to train the MIMLLN classifier. Through these experiments, we determined that a batch size of 16 and dropout rate of 0.5 for every layer yielded optimal performance. The activation function ReLU was used in model. We trained the model for 15 epochs, employing learning rate of le-3 and cross-entropy as the loss function. To prevent overfitting, we used the Adam optimizer with a weight decay of 1e-3. For the Reviewer Dis- agreement Classifier, we trained the models using true Sentiment Disparity Aspect pairs with true as- pect and sentiment labels. We use a batch size of 16, maximum length of 280 tokens, and dropout probability of 0.1. The Adam optimizer was em- ployed with learning rate of 1e-5. All models were trained on an NVIDIA A100 40GB GPU.
B More details on Aspect Sentiment Model
MIMLLN for Sspect-Category Sentiment Anal- ysis (ACMIMLLN) operates on the assumption that the sentiment of a mentioned aspect category in sentence aggregates the sentiments of words that indicate that aspect category. In MIMLL, words that indicate an aspect category are termed "key instances' of that category. Specifically, AC- MIMLLN comprises two components: an attention- based aspect category detection (ACD) classifier and an aspect-category sentiment analysis (ACSA) classifier. Given a sentence, the ACD classifier, as an auxiliary task, assigns weights to the words for each aspect category. These weights signify the likelihood of the words being key instances of their respective aspect categories. The ACSA clas- sifier initially predicts the sentiments of individual words. then determines the sentence-level sen- timent for each aspect category by integrating the respective weights with the word sentiments. The
D Result of Aspect Sentiment Classifier
We present the results of the Aspect Sentiment Model in Table 3. We conducted experiments us- ing various settings of BERT transformer layers, such SCIBERT (Beltagy et al., 2019). SCIBERT "https://huggingaaae.c

---

Aspect-Category Model Scores
MOT
CLA
SOU
SUB
MEA
ORI
REP
Fl laap
0.62
0.67
0.69
0.70
0.72
0.64
0.55
(Kumar al., 2021)
F1 sent
0.69
0.70
0.68
0.71
0.52
0.62
0.59
F1 ass
0.81
0.88
0.78
0.76
0.83
0.86
0.73
MIMLLN (BERT)
Fl
0.78
0.86
0.76
0.74
0.81
0.83
0.71
F Lasp
0.80
0.86
0.77
0.75
0.83
0.86
0.71
MIMLLN(ROBERTa)
F1 sent
0.76
0.84
0.74
0.73
0.82
0.81
0.69
F Flasy Laa
0.79
0.86
0.77
0.73
0.82
0.85
0.68
MIMLLN(ScIBERT)
F1 sent
0.77
0.85
0.75
0.72
0.81
0.81
0.67
F Plasp
0.75
0.86
0.77
0.73
0,81
0.86
0.68
MIMLLN(SPECTRE)
sent
0.74
0.85
0.76
0.73
0.80
0.84
0.66
Table 3: Performance of all the models and baselines from the experiments. Here, (Flasp represents the aspect- category scores, I'l sent represents sentiment-category scores)
is pre-trained language model based on BERT and trained on large-scale, labeled scientific dataset. Additionally, experiment with SPECTER (Co- han al., 2020), which is a model designed for learning representations of scientific papers. This model based on a Transformer language model pre-trained on citations. We found that the model performed optimally with BERT for aspect classifi- cation across all aspect categories.
ing contradictions. We took single review pair at time to detect the contradiction. The structure of this prompt is shown in the Figure 5
Here, {review1] } and (review2} are placehold- ers that represent the text from pair of reviews for which we aim to identify contradictions.
During our testing, we identified many false pos- itive cases in the output of ChatGPT. For exam- ple, when presented with the reviews (paper id:
Furthermore, compared our results with those of aspect and sentiment multi-task model (Ku- mar et al., 2021). This model utilizes shared BERT transformer layer and employs different feed-forward networks on top as task-specific lay- ers. For each aspect category, pretrained BERT performed the best (RoBERTa performed the same for meaningful comparison).
R1: The argument of biological plausibility is not justified
R2: Moreover the biological plausibility that is used as argument at several places seems to be false advertising in my view
Regarding aspect-based sentiment, BERT per- formed the best for the aspects of Motivation, Clar- ity. Substance, and Replicability. However, for the Soundness category, SPECTER performed bet- ter. As for the Meaningful Comparison aspect, RoBERTa showed better performance.
ChatGPT incorrectly labeled these reviews as contradictions. However, our baseline system accu- rately identified that these statements do not present contradiction.
Additionally, ChatGPT sometimes compares two distinct aspects. For example:
E Comparsion with ChatGPT
We conducted comparative study between our pro- posed baseline model and ChatGPT and BARDS. focusing on multi-turn dialogue inputs. We used the OpenAI API for the task. To evaluate the ef- fectiveness our model, we experimented with various prompts, selecting the most effective one. The prompt chosen is particularly adept at extract-
RI: This paper is not well-written.
R2: The results reasonable and significant.
In this instance, the first reviewer is commenting on the clarity of the paper, while the second re- viewer is commenting the substance of the paper. These represent different perspectives. Chat- GPT occasionally fails in such scenarios. However, our proposed baseline incorporates an intermediate aspect sentiment model, which helps reduce these types of errors.
'https://bal www.ardd.oole..ooo hhttps://platfort openai. com/docs/ api-reference/

---

You are helpful assistant.
System
Could you please evaluate the reviews labelled RI and R2? interested finding whether they present conflicting viewpoints. they do, could identify the specifie comments from each review that create this contradiction?
User
Reviewl: (reviewl)
Review2: (review2)
Figure 3: ChatGPT input format
F Error Analysis
and the simplicity of the method." For this com- ment, the predicted aspect category by the model 'motivation'. These two comments provide con- trasting viewpoints on the aspect of 'mooivation' Yet, due the misclassification by the aspect sen- timent model, this discrepancy isn not flagged by the second model.
We conducted analysis of the predictions made by our proposed baseline to identify the areas where it most frequently fails.
F.1 Lack of Complete Review Context:
Our proposed baseline can sometimes fail in accu- rately predicting contradictions when the sentences are significantly related to preceding comments within the same review.
Take, for instance, the following comment: "This is elegant intuitive algorithm that, to my knowl- edge, has not appeared in previous literature."
In this case, the model incorrectly predicts con- tradiction with another review comment that dis- cusses different algorithm negatively. The issue arises from the model's inability to distinguish be- tween the algorithm discussed in this comment and a different algorithm mentioned negatively by an- other reviewer. While prediction the model needs to learn from previous discussions well. Such an approach can be considered potential area for future work.
F.2 Issues with Lengthy Comments:
Our proposed baselines sometimes stumble in pre- dicting the right contradictions when dealing with particularly long sentences or complex review com- ments (significant amount of technical terms or mathematical symbols) which can lead to confu- sion.
F.3 Error Propagation
We found error propagation from the first model to the second. To illustrate, consider the following example: Reviewer I's comment reads: "While it is very interesting apply adversarial noise in real data, this approach is clearly motivated explained." The Aspect sentiment model pre- diets the aspect for this comment as 'soundness' but misses out on 'motivation'. Reviewer 2's com- ment states: "In overall, liked its clear motivation


Reviews:
[{"review_id": "pW9SCPjpEw", "reviewer": "Reviewer_z23x", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Query: The main concern I have for this paper is the problem definition is not clear. Review contradictions naturally exist and should not be regarded as a good or bad thing, so what is the conclusion of a paper that has fewer contradictions? The conclusion is subjective and individual. For example, other ethical problem, such as the Fairness problem, has a soundness motivation supported, since unfairness is definitely not good. And The new task proposed in this paper is not well-stated. Is there any solution for this task?\n\nResponse: Thank you for your constructive feedback on our paper. We appreciate your concerns regarding the clarity of the problem definition and the potential subjectivity of identifying contradictions in reviews. Allow me to address these concerns comprehensively.\nWe intend not to label contradictions as inherently 'good' or 'bad.' Instead, we aim to provide editors with a system that draws attention to potential reviewer inconsistencies. The primary motivation behind our system is to aid editors, given their often demanding roles and the multitude of decisions they face. By highlighting contradictions, our system facilitates editors in pinpointing areas of discussion, enabling them to delve deeper into the nuances and make well-informed decisions. It's essential to reiterate that our system is designed solely for internal usage by editors and is not intended for authors or reviewers.\nConsider an example: if one review suggests, \"the paper lacks new findings,\" while another states, \"the paper demonstrates novelty,\" these might be perceived as contradictory. But the essence lies in understanding the context and intent of these comments. Our system brings such discrepancies to the forefront, allowing editors to analyse, discuss and resolve them based on established guidelines.\n\nRegarding the new task proposed, the paper introduces a fresh perspective. While our exposition could be more precise. In conclusion, our contribution aligns well with the ethos of EMNLP. The goal is to add clarity to the review process, ensuring that the final decisions made by editors are as informed and nuanced as possible. We appreciate your feedback and will work towards enhancing the clarity and precision of our paper by adding the above points. We hope you reconsider its fit upon our revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles a longstanding issue in the peer review process, focusing on the identification of contradictions among reviewers' comments. This problem has persisted over time and significantly impacts the efficacy of the review process. The authors not only shed light on this important problem but also enhance the transparency of their work by sharing their dataset and code, facilitating further research and development in the field.", "reasons_to_reject": "1. The main concern I have for this paper is the problem definition is not clear. Review contradictions naturally exist and should not be regarded as a good or bad thing, so what is the conclusion of a paper that has fewer contradictions? The conclusion is subjective and individual. For example, other ethical problem, such as the Fairness problem, has a soundness motivation supported, since unfairness is definitely not good. \n\n2. And The new task proposed in this paper is not well-stated. Is there any solution for this task? \n\nSo I think this paper is not appropriate for EMNLP because a task should be undisputed and clear.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly focuses on NLP engineering experiments. This paper addresses the pivotal role of peer review in scientific publishing, highlighting its impact on editorial decisions. The authors present ContraSciView, a new dataset containing review pairs with contradictions extracted from ICLR and NeurIPS conference reviews, totaling around 8.5k papers and 28k review pairs. They also introduce a baseline model for automatically detecting contradictory comments within review pairs, shedding light on the complexities and prospects of identifying contradictions in peer reviews.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "peWKlUUdX0", "reviewer": "Reviewer_H8L1", "summary": "", "questions": "Question A: Could you clarify the distinctions between a review, a review pair, and a review pair comment?\n\nQuestion B: I'm interested in understanding the average, minimum, and maximum number of reviews per paper. It would be beneficial to include more statistical details about the dataset. Can you add such statistics? \n\nQuestion C: In Figure 2, you display seven aspects of paper reviews classified as contradictions or not. Do all these aspects have corresponding comments, or are they solely multiple-choice fields? For instance, if \"soundness\" involves multiple-choice responses, how did you identify disagreements?\n\nQuestion D: How frequently did the annotators encounter instances where they were unable to determine the presence of a contradiction?\n\nQuestion E: You mentioned splitting the dataset into three parts: 80% for training, 10% for validation, and 10% for testing. Could you elaborate on the choice of a 10% testing set? Was cross-validation employed?\n\nQuestion F: I'm curious about the specifics of how ChatGPT was employed. Was the API used? Did you utilize the complete set of review pairs or a subset?\n\nQuestion G: Given your reference to earlier systems for detecting contradictions, would it not be insightful to compare your results with those systems rather than solely testing various models?\n\nQuestion H: Why was Google BARD not tested?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Query 1: The positive aspect of automating the peer-review process and encouraging reviewer consensus could potentially turn into a negative if editors and meta-reviewers overly depend on the system. This may lead them to accept or reject papers without thoroughly reviewing the comments, raising ethical concerns and violating the ACM Code of Ethics and Professional Conduct, the principle of avoiding harm to the community https://www.acm.org/code-of-ethics.\n\nResponse: We sincerely appreciate the concerns about the potential over-reliance on an automated peer-review process. Automation is intended to aid and streamline the review process but should not replace the diligent, thoughtful examination traditionally expected of editors and meta-reviewers. We recognize the pivotal role of human judgement, especially in nuanced situations where context and detailed understanding are imperative. \n\nThe essence of peer review is thoroughness and integrity, and the ACM Code of Ethics and Professional Conduct underscores the importance of avoiding harm to the community. Hence, while our system seeks to encourage reviewer consensus and improve efficiency, it is paramount that it complements, not supplants, human discernment. We endorse the principle that any tool, automated or otherwise, should be used judiciously, ensuring ethical guidelines and the spirit of peer review remain uncompromised. We will emphasize this perspective in our work to ensure the academic community retains the sanctity of the peer-review process.\n\nThank you for your constructive feedback regarding our paper. We appreciate your concerns and would like to address them in detail. We understand the ethical implications of the potential inaccuracies of the system. We intend to develop and promote responsible use of our findings, ensuring that users are aware of its limitations. To address this concern more comprehensively, we will expand the Ethical Concerns section to discuss potential misuse, the ramifications of inaccuracies, and the measures we've taken or proposed to mitigate these concerns. A more in-depth discussion will provide readers with a broader perspective on the matter. We will add the following in the ethical section:-\n\nWill our system sometimes generate incorrect contradictions or overlook certain contradictions? Yes, it will. Like other general AI models, this model is not 100% accurate. Editors or Chairs might rely on more than just the contradictions predicted by the model to make decisions. It is important to emphasize that the primary purpose of this system is to assist editors by highlighting potential contradictions between reviewers; this system is only for editors' internal usage, not for authors or reviewers. Especially given their often busy schedules and the myriad of decisions they must make. The contradictions among the reviewers will help editors to identify and initiate a discussion to resolve the confusion between the reviewers and to make an informed decision.\n\nSince reviews are frequently extensive and intricate, it's challenging for editors to scrutinize every comment and address conflicts. This system aims to aid them in spotting such contradictions. For instance, if one comment reads, \"The paper doesn't have any new findings,\" and another reviewer mentions, \"The paper is somewhat novel,\" the editor might not immediately perceive this as contradictory. However, understanding the context and intent behind these comments is vital. If they represent a contradiction, editors can address it using the standard guidelines.\n\nDoes the system detect all contradictions? No. If the system fails to identify a contradiction, it doesn't automatically mean none exists. Given its nature as a general AI system, there's a possibility of it presenting false negatives. Editors relying solely on this system could impact the review process. Any contradictions spotted outside the system's recommendations should be addressed according to the guidelines.\n\nWe aim to add these points and thoroughly analyze the ethical concerns surrounding our research. \nThank you for pointing to this critical aspect. We're committed to promoting responsible use of our system in alignment with professional ethics.\n\nQuery 2: The dataset used may exhibit labelling issues. Not all differing comments necessarily contradict each other; some may complement each other. Furthermore, the severity of the contradiction is not considered, which could be addressed through categories like high, average, and low levels of contradiction\n\nResponse: Thank you for your comment. We acknowledge that not all differing comments inherently exhibit contradiction; some might be complementary. We have added this in the limitation section of the paper. \nExplicit Contradictions: These are clear, direct, and unmistakable contradictions that can be easily identified. For example:\nThe author claims in the introduction that \"X has been proven to be beneficial,\" but in the discussion section, they state that \"X has not been shown to have any benefit.\"\nOne reviewer says, \"Figure 2 clearly shows the relationship between A and B,\" while another reviewer comments, \"Figure 2 does not show any clear relationship between A and B.\"\nImplicit Contradictions: These are more subtle and may require deeper understanding or closer examination to identify. It may require the annotators to read the paper and also learn many related works or things to annotate. They are not directly stated but can be inferred from the context or how information is presented. For example:\nReview 1: \"the method lacks algorithmic novelty and the exposition of the method severely inhibits the reader from understanding the proposed idea .\"\nReview 2: \"the work presented is novel but there are some notable omissions - there are no specific numbers presented to back up the improvement claims graphs are presented but not specific numeric results - there is limited discussion of the computational cost of the framework presented - there is no comparison to a baseline in which the additional learning cycles used for learning the embedding are used for training the student model .\"\n For instance, Review1 mentions that the method lacks algorithmic novelty, and Review2  acknowledges the work as a novel but points out some notable omissions. This difference in perception is not a direct contradiction, as one reviewer finds the method lacking in novelty, while the other recognizes it as novel but with some omissions.\nYour suggestion regarding grading contradictions based on severity – high, average, or low – is insightful. However, classifying them into three finely delineated categories would require more extensive training for the annotators. The annotation process is already time-consuming and complex, so we decided to keep it as simple as possible for the initial stage. Nevertheless, we see merit in this categorization. While this study serves as foundational work, we aim to integrate such granularity in the future. We will include and elaborate on this in the 'Future Work' section of the paper. Thank you for your valuable feedback.\nQuery 3: The issue of detecting disagreement among reviewers is not new and has been explored before, making the contribution less clear.\n\nResponse: While we recognize that the detection of disagreements among reviewers in broader domains, such as web sources and customer reviews, is not a novel concept, our research delves into a very specific niche: contradictions within peer reviews. To our knowledge, this specific aspect remains largely untouched in academic studies. It's essential to understand that how peer reviews are penned is inherently different from other review types. This unique writing style, combined with the untouched nature of this research area, positions our dataset and study as both pivotal and essential.\n\nQuery 4: Cohen's kappa of 0.62 is not excellent.\n\nResponse: This particular annotation requires reading the entire review and understanding the scientific information very carefully. The task is inherently more challenging as compared to other simple annotation tasks. Given the inherently challenging nature of this task, even minor discrepancies in interpretation or understanding can lead to disagreements. This is not a reflection on the competency of our annotators; in fact, they are highly qualified. Instead, it underscores the depth and intricacy of the task at hand.\nThus, the kappa score we've achieved, while not perfect, is understandable. It's reasonably lower because the task is unreasonably more complex than most. \n\nQuestions For The Authors:\n\nQuestion A: Could you clarify the distinctions between a review, a review pair, and a review pair comment?\n\nResponse: A 'review' is a collection of comments/sentences written by one reviewer. Think of it like a list: {comment1, comment2, comment3, ...}.\n\nA 'review pair' is like taking two of these lists - one from Reviewer1 and one from Reviewer2. It looks like this: {Reviewer1's comments, Reviewer2's comments}.\n\nLastly, a 'review pair comment' picks one comment from each reviewer and pairs them together. So, it's a set of pairs: {(comment from Reviewer1, comment from Reviewer2), ...}.\n\nQuestion B: I'm interested in understanding the average, minimum, and maximum number of reviews per paper. It would be beneficial to include more statistical details about the dataset. Can you add such statistics?\nResponse: We appreciate the suggestion of the reviewer. We recognize the importance of providing comprehensive statistical details regarding our dataset. Understanding the average, minimum, and maximum number of reviews per paper can offer valuable insights into the depth and breadth of our data. We will ensure to include these specifics using a pie chart, along with any other relevant statistics, in the updated version of our paper. \n\nQuestion C: In Figure 2, you display seven aspects of paper reviews classified as contradictions or not. Do all these aspects have corresponding comments, or are they solely multiple-choice fields? For instance, if \"soundness\" involves multiple-choice responses, how did you identify disagreements?\n\nResponse: If \"soundness\" involves multiple-choice responses, multiple review pair comments will pass through the classifier for prediction.  For example {( soundness first comment from Reviewer1, soundness first comment from Reviewer2), ( soundness first comment from Reviewer1, soundness second comment from Reviewer2) ...}.\n\nQuestion D: How frequently did the annotators encounter instances where they were unable to determine the presence of a contradiction?\nResponse: Around avg 8% of the time, annotators could not find the presence of contradiction. The experts then resolved it. \n\nQuestion E: You mentioned splitting the dataset into three parts: 80% for training, 10% for validation, and 10% for testing. Could you elaborate on the choice of a 10% testing set? Was cross-validation employed?\nResponse:  We randomly split the data into Training, Testing, and Validation sets. The decision to split the dataset into 80% training, 10% validation, and 10% testing is grounded in a traditional approach to dataset division, aiming to maximize the amount of data available for training while still retaining separate portions for model validation and testing. \n\nQuestion F: I'm curious about the specifics of how ChatGPT was employed. Was the API used? Did you utilize the complete set of review pairs or a subset?\nResponse: We used the OpenAI API for the task. We took a single review pair at a time to detect the contradiction. We have provided a more detailed analysis and prompts used in Appendix A.3 and Figure 5 in the paper.\n\nQuestion G: Given your reference to earlier systems for detecting contradictions, would it not be insightful to compare your results with those systems rather than solely testing various models?\nResponse: Thank you for your feedback. While we have referenced earlier systems for detecting contradictions, our primary focus was on the recent advancements in Natural Language Inference, especially those with contradiction labels. These provide a more pertinent benchmark for our research. That said, we have directly compared our results with those from pre-trained models. We aim to provide a comprehensive understanding of how our approach stands compared to the latest in the field, but we acknowledge the value of juxtaposing our results with earlier systems for a broader perspective. We will add those comparisons as well in our results.\n\nQuestion H: Why was Google BARD not tested?\nResponse: Thank you for bringing up Google BARD. We apologize for the oversight. We recognize the importance of comparing our results with notable models like Google BARD.  We used the BARD API (https://github.com/dsdanielpark/Bard-API) for our evaluation. We provided the same prompt to both Google BARD and CHATGPT for a fair comparison. We compared our findings with those achieved by Google BARD. BARD scored an F1 of 61.35 on the test set, approximately 12 points lower than our baseline model. This is likely due to its need for more explicit training for this downstream task. We found that Google BARD registered more false positives than CHATGPT. We will include a detailed analysis of this in the updated version of our paper.\n\nMissing References:\nQuery 1: Please check all references: Anne Borcherds and Managing Editor. 2017. How to deal with conflicting reviewer comments.\nResponse: Thank you for highlighting the reference by Anne Borcherds and the Managing Editor from 2017. We will ensure to incorporate this into our paper. Additionally, we'll undertake a thorough review to ensure no other crucial references have been overlooked. We appreciate your diligence in pointing this out and aiding in the enhancement of our work.\n\nTypos Grammar Style And Presentation Improvements:\n\nQuery 1: The statement \"We generated a list of review comment pairs for the remaining pairs...\" lacks clarity about what is meant by \"remaining pairs.\"\n\"This labeling process resulted in 17,095\". This doesn't match the numbers in Table 1. \n\nResponse: We apologize for the confusion due to ambiguous distinctions between a review, a review pair, and a review pair comment. Table 1 presents the statistics for review pairs and the total number of reviews. Among the total review pairs listed in Table 1, some might not include review pair comments that share the same aspect but have conflicting sentiments, suggesting no potential contradictions. Such pairs can be immediately deemed non-contradictory. Therefore, manual labelling was required for only the remaining 17,095 review pairs for some of them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This research addresses the growing number of submitted manuscripts, potentially automating parts of the peer-review process and promoting reviewer agreement.\n\nThe generated dataset holds practical value for the community.\n\nThe paper's content is easy to understand.", "reasons_to_reject": "The positive aspect of automating the peer-review process and encouraging reviewer consensus could potentially turn into a negative if editors and meta-reviewers overly depend on the system. This may lead them to accept or reject papers without thoroughly reviewing the comments, raising ethical concerns and violating the ACM Code of Ethics and Professional Conduct, the principle of avoiding harm to the community https://www.acm.org/code-of-ethics. \n\nThe dataset used may exhibit labeling issues. Not all differing comments necessarily contradict each other; some may complement each other instead. Furthermore, the severity of the contradiction is not considered, which could be addressed through categories like high, average, and low levels of contradiction.\n\nThe issue of detecting disagreement among reviewers is not new and has been explored before, making the contribution less clear. \n\nCohen’s kappa of 0.62 is not excellent.", "questions_to_authors": "Question A: Could you clarify the distinctions between a review, a review pair, and a review pair comment?\n\nQuestion B: I'm interested in understanding the average, minimum, and maximum number of reviews per paper. It would be beneficial to include more statistical details about the dataset. Can you add such statistics? \n\nQuestion C: In Figure 2, you display seven aspects of paper reviews classified as contradictions or not. Do all these aspects have corresponding comments, or are they solely multiple-choice fields? For instance, if \"soundness\" involves multiple-choice responses, how did you identify disagreements?\n\nQuestion D: How frequently did the annotators encounter instances where they were unable to determine the presence of a contradiction?\n\nQuestion E: You mentioned splitting the dataset into three parts: 80% for training, 10% for validation, and 10% for testing. Could you elaborate on the choice of a 10% testing set? Was cross-validation employed?\n\nQuestion F: I'm curious about the specifics of how ChatGPT was employed. Was the API used? Did you utilize the complete set of review pairs or a subset?\n\nQuestion G: Given your reference to earlier systems for detecting contradictions, would it not be insightful to compare your results with those systems rather than solely testing various models?\n\nQuestion H: Why was Google BARD not tested?", "paper_topic_and_main_contributions": "In this paper, the authors introduce a model designed to detect contradictions in reviewers' feedback. This model is helpful for editors, enabling them to pinpoint contradictory reviews swiftly, thus streamlining the review assessment process and saving time. The proposed model outperforms existing text classification models in its performance. To train the model, the authors curated a dataset comprising pairs of reviews, annotating conflicts to facilitate the training of machine learning algorithms.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please check all references: \nAnne Borcherds and Managing Editor. 2017. How to deal with conflicting reviewer comments."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The statement \"We generated a list of review comment pairs for the remaining pairs...\" lacks clarity about what is meant by \"remaining pairs.\"\n\n“This labeling process resulted in 17,095”. This doesn’t match the numbers in Table 1.", "Justification_for_Ethical_Concerns": "The positive aspect of automating the peer-review process and encouraging reviewer consensus could potentially turn into a negative if editors and meta-reviewers overly depend on the system. This may lead them to accept or reject papers without thoroughly reviewing the comments, raising ethical concerns and violating the ACM Code of Ethics and Professional Conduct, the principle of avoiding harm to the community https://www.acm.org/code-of-ethics."}}, {"review_id": "S14S9opLyF", "reviewer": "Reviewer_WLX5", "summary": "", "questions": "Q1: The authors use “review pairs” and “review comment pairs” interchangeably. Which is it exactly? How are the reviews chuncked into review comments? Is each review comment processed by the reviewer disagreement classifier individually regardless of the context?\n\nQ2: How are the aspects and sentiments classified in the dataset preprocessing step (line 145 to line 157)? Do the aspects come from the ASAP-Review dataset? Are the authors aware that they are not ground truth labeled by humans but rather predictions made by an aspect tagger?\n\nQ3: Similarly, what data is the aspect sentiment model trained on? Why is the the results from this step not presented and analyzed? Do you consider error propagation from the first model to the second?\n\nQ4: What are the respective definitions of implicit/explicit contradictions. The authors mention that they “do not delve into implicit contradictions” in the limitations section. However, these concepts are not mentioned in the annotation guidelines. This might have been a source of inter-annotator disagreement.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The authors use “review pairs” and “review comment pairs” interchangeably. Which is it exactly? How are the reviews chuncked into review comments? Is each review comment processed by the reviewer disagreement classifier individually regardless of the context?\nResponse: Suppose the review contains two reviews, say R1 and R2. R1 and R2 can have 100,50 sentences each. We call each such pair of sentences for R1 and R2 as review comment pairs. The review pair is full text, and the review pair comment is at sentence level. \n\nYes, our proposed classifier doesn’t take the full review context while classifier. However, it beats the zero-shot large language models like CHATGPT. We encourage a system that considers full context while classifying future work, which will also be mentioned in the Future Work section. We have also mentioned this in Appendix error analysis A4.1.\n\nQ2: How are the aspects and sentiments classified in the dataset preprocessing step (line 145 to line 157)? Do the aspects come from the ASAP-Review dataset? Are the authors aware that they are not ground truth labeled by humans but rather predictions made by an aspect tagger?\nResponse: Yes, we understand that the ASAP-Review dataset contains predictions labelled by an aspect tagger on a human-annotated label. We used those human-annotated labels to train our classifier. We will clarify this in the paper.\nQ3: Similarly, what data is the aspect sentiment model trained on? Why is the the results from this step not presented and analyzed? Do you consider error propagation from the first model to the second?\nResponse: Trained on ASAP-Review dataset. Due to the page limit of the paper, we have added the result in the Appendix of the paper, A.2. Yes, there will be error propagation from the first model to the second.  To illustrate, consider the following examples:\nReviewer 1’s comment reads: “While it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained.” The Aspect sentiment model predicts the aspect for this comment as 'soundness' but misses out on 'motivation'.\nReviewer 2’s comment states: “In overall, I liked its clear motivation and the simplicity of the method.” For this comment, the predicted aspect category by the model is 'motivation'.\nThese two comments provide contrasting viewpoints on the aspect of 'motivation'. Yet, due to the misclassification by the aspect sentiment model, this discrepancy isn't flagged by the second model. \nWe thank the reviewer for pointing this out; we will add this analysis in more detail in the Error analysis section of the Appendix.\n\nQ4: What are the respective definitions of implicit/explicit contradictions. The authors mention that they “do not delve into implicit contradictions” in the limitations section. However, these concepts are not mentioned in the annotation guidelines. This might have been a source of inter-annotator disagreement.\n\nResponse: Explicit Contradictions: These are clear, direct, and unmistakable contradictions that can be easily identified. For example:\nThe author claims in the introduction that \"X has been proven to be beneficial,\" but in the discussion section, they state that \"X has not been shown to have any benefit.\"\nOne reviewer says, \"Figure 2 clearly shows the relationship between A and B,\" while another reviewer comments, \"Figure 2 does not show any clear relationship between A and B.\"\n\nImplicit Contradictions: These are more subtle and may require deeper understanding or closer examination to identify. It may require the annotators to read the paper and also learn many related works or things to annotate. They are not directly stated but can be inferred from the context or the way information is presented. For example:\n\nReview 1: “the method lacks algorithmic novelty and the exposition of the method severely inhibits the reader from understanding the proposed idea .”\nReview 2: “the work presented is novel but there are some notable omissions - there are no specific numbers presented to back up the improvement claims graphs are presented but not specific numeric results - there is limited discussion of the computational cost of the framework presented - there is no comparison to a baseline in which the additional learning cycles used for learning the embedding are used for training the student model . \"\n\n For instance, Review1 mentions that the method lacks algorithmic novelty, and Review2  acknowledges the work as a novel but points out some notable omissions. This difference in perception is not a direct contradiction, as one reviewer finds the method lacking in novelty, while the other recognizes it as novel but with some omissions.\n\nAlso, if the reviewer mentions some reference to the paper, such as looking at section 3 of the paper, it needs to be clarified, while another reviewer says I find the explanation of the use of contrastive learning we explained. However, section 3 talks about contrastive learning, so both are referring to the same particular. However, it requires reading the paper to determine the contradiction. \nWe mentioned in the Annotator Guidelines that “you may find more detailed annotation guidelines on our shared repository”. However, we will also add this point in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The application of NLP to the peer reviewing process remains an understudied topic that deserves more attention. There is a need to encourage the creation of resources focused on  further exploring this area.", "reasons_to_reject": "1. The overall motivation and idea is well described. However, there are crucial details missing in the methodology description, as indicated in the questions raised for the authors. These omissions raise concerns about the paper's soundness and validity.\n\n2. The dataset is crawled from OpenReview without obtaining explicit consent from the paper authors and reviewers. This raises potential ethical and legal issues that could have long-term implications.\n", "questions_to_authors": "Q1: The authors use “review pairs” and “review comment pairs” interchangeably. Which is it exactly? How are the reviews chuncked into review comments? Is each review comment processed by the reviewer disagreement classifier individually regardless of the context?\n\nQ2: How are the aspects and sentiments classified in the dataset preprocessing step (line 145 to line 157)? Do the aspects come from the ASAP-Review dataset? Are the authors aware that they are not ground truth labeled by humans but rather predictions made by an aspect tagger?\n\nQ3: Similarly, what data is the aspect sentiment model trained on? Why is the the results from this step not presented and analyzed? Do you consider error propagation from the first model to the second?\n\nQ4: What are the respective definitions of implicit/explicit contradictions. The authors mention that they “do not delve into implicit contradictions” in the limitations section. However, these concepts are not mentioned in the annotation guidelines. This might have been a source of inter-annotator disagreement.", "paper_topic_and_main_contributions": "The topic of this paper is the detection of conflicts/disagreements between different reviewers for the same paper in scientific peer reviews. The main contribution is the formulation of this task and the release of ContraSciView, an annotated dataset of 50,303 pair of review comments. The authors also propose a baseline model to perform this task and compare it to zero-shot ChatGPT. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LbS9CRDLA6", "reviewer": "Reviewer_B2WG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and pointing out these formatting and typographical issues in the manuscript. We sincerely appreciate your attention to detail. Here's our response to each of the mentioned concerns:\n\nP2, l112: Thank you for pointing out the missing space. It will be corrected to \"Gong et al., 2018). Well-known...\"\n\nP3, description Figure 2: We apologize for the oversight. Spaces will be added to make it 'y axis: count', 'c: contradiction', and 'n: non-contradiction'.\n\nP3, Figure 3: The inconsistency between the figure and its description was a typo. We will correct the figure to display 'ASOP' for clarity.\n\nP3, Figure 3: Your suggestion to include (1) and (2) in the Figure to match the description is appreciated. We will make these amendments to enhance clarity.\n\nP3, l236: Thank you for spotting the missing space. It will be corrected to 'Pair (SDAP)'.\n\nP4, l241: The dash will be removed and the text will be updated to \"as follows:\".\n\nP4, l242-l245: We understand your concerns regarding the sentence's grammar. We will review and rephrase this section to ensure better clarity.\n\nP4, l245-l247: We acknowledge your observation. The sentence will be revised to more clearly convey that a single review can encompass multiple aspects and sentiments.\n\nP4, Table 2: We apologize for the oversight in the spacing. The text will be corrected to 'Finetuned on (ANLI+ALL)'.\n\nP4, Table 2: The missing space in the description will be corrected to 'R: Recall'.\n\nP5, l340: Thank you for pointing out the grammatical error. The sentence will be corrected to \"We have utilized the open...\"\n\nP9, l777: The missing space will be rectified to \"... with the reviews (paper id)...\".\n\nP10, l826: We will correct the spacing error to \"comments (significant amount...\".\n\nIn summary, we appreciate your constructive feedback, which will certainly enhance the quality and clarity of our paper. All mentioned issues will be addressed in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Code and dataset is publicly available\n- Approach might be interesting for other NLI tasks\n", "reasons_to_reject": "None.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an approach to automatically detect contradictions in reviews for scientific papers. To this end, the authors manually annotated a corpus which in turn was used for the training and evaluation of a baseline system for this novel task. \n\nThe main contributions are:\n- Introduction of the novel task of automatically identifying contradictions between reviewers' comments on the same paper\n- Creation of a novel dataset containing 50k pairs of review comments which were extracted from reviews of about 8.5k papers\n- Establishment of a baseline method as a reference point for further research\n\nMain strengths of this paper are the dataset and the evaluation.\n1. For the creation of the dataset, the authors collected reviews of the ICLR and NeurIPS conferences. As any other conference, there are already pre-built aspect categories (Motivation, Soundness, Clarity, ...) along with their sentiment. Therefore, a contradiction is located when the reviewers mention the same aspect but with opposing sentiment. These contradiction review comment pairs are then manually annotated. \n2. The resulting dataset is used as input to train a baseline system implemented as a pipeline: \n(i) Training of an Aspect Sentiment Model to classify the aspects and the sentiment of each review comment. As framework, the authors use the Multi-Instance Multi-Label Learning Network (MIMLLN) of Li et al. (2020) with four models. Based on the outcome of MIMLLN, the authors identify so called Sentiment Disparity Aspect Pairs (SDAP), i.e., pairs mentioning the same aspect but differ in sentiment. Those SDAPs are the input for the next step. \n(ii) Training of a Reviewer Disagreement Classifier based on NLI techniques. Unlike in traditional NLI tasks, the output categories are mapped to 'contradiction' and 'non-contradiction' in accordance with the task at hand. As models, the authors used DistilBERT, XLNet base and large as well as RoBERTa base and large. Evaluation shows that RoBERTa large performs best. \n3. For completion reason, the authors evaluated the models from (ii) by training them on the ANLI+ALL dataset. Test was performed on the test set from their own dataset. Results show that better scores are obtained when trained on their own dataset. The authors attribute this to the fact that reviews contain a more technical style of writing than e.g. Wikipedia entries. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "P2, l112 Missing space 'Gong et al., 2018).Well-known...\nP3, description Figure 2: Missing space 'y axis:count', 'c:contradiction', 'n:non-contradiction'\nP3, Figure 3: The Figure says 'AOS?' (Purple prism) but description says 'ASOP'\nP3, Figure 3: There should be a (1) and (2) in the Figure to match the description \nP3, l236: missing space 'Pair(SDAP)'\nP4, l241: remove dash (as follows:)\nP4, l242-l245: This sentence seems ungrammatical\nP4, l245-l247: However, it has not considered that a review can have two different aspects and sentiment.\nP4, Table 2: Missing space in second row 'Finetuned on(ANLI+ALL)'\nP4, Table 2: Missing space in description 'R:Recall'\nP5, l340: 'We have utilizeD the open...'\nP9, l777: Missing space '... with the reviews(paper id)...'\nP10, l826: Missing space 'comments(significant amount...'"}}]

Author rebuttal:

Meta-review: This paper has got mixed reviews and has also resulted in quite a long and detailed rebuttal.
The content of the paper is, no doubt, interesting and forms a solid short paper which can spur a lot of discussions during the conference (which is always encouraging). Despite having some flaws in the experiment setup, this paper should still make it to the conference.

Now coming to the ethical side of things, the paper has also raised some ethical concerns, and the ethics meta-reviewer has listed down a set of points that the paper must address before it can be published. I remind the authors to carefully take all these into account to prepare the camera-ready version.
Decision: Accept-Main