Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals
Authors: Sukannya Purkayastha
Anne Lauscher
Iryna Gurevych
Venue: EMNLP 2023 Main (2023)
Keywords: Peer Reviews
Rebuttals
Attitude Roots
Jiu-Jitsu Persuasion
TL;DR: Using etstablished concepts in peer reviews, it is possible to map different reviews to clusters. These clusters can then be mapped to generic rebuttals.

Abstract: In many domains of argumentation, people’s arguments are driven by so-called attitude roots, i.e., underlying beliefs and world views, and their corresponding attitude themes. Given the strength of these latent drivers of arguments, recent work in psychology suggests that instead of directly countering surface-level reasoning (e.g., falsifying the premises), one should follow an argumentation style inspired by the Jiu-Jitsu ``soft'' combat system: first, identify an arguer's attitude roots and themes, and then choose a prototypical rebuttal that is aligned with those drivers instead of trying to invalidate those. In this work, we are the first to explore Jiu-Jitsu argumentation for peer reviews by proposing the novel task of attitude and theme-guided rebuttal generation. To this end, we enrich an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. To facilitate this process, we recast established annotation concepts from the domain of peer reviews (e.g., aspects a review sentence is relating to) and train domain-specific models. We then propose strong rebuttal generation strategies, which we benchmark on our novel dataset for the task of end-to-end attitude and theme-guided rebuttal generation and two subtasks.

Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals
Sukannya Purkayastha' Anne Lauscher Iryna Gurevych'
Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.Al), Technical University of Darmstadt
Data Science Group, University of Hamburg www.ukp tu Su-darmstadt.cc de
Abstract
Review
Itink this interesting direction, but the present paper unolear.
many domains of argumentation, people's s ar- guments are driven by so-called attitude roots, ie.. underlying beliefs and world views, and their corresponding attitude themes. Given the strength of these latent drivers of arguments, re- cent work in psychology suggests that instead of directly countering surface-level reasoning (e.g.. falsifying given premises), one should follow an argumentation style inspired by the Jiu-Jitsu "soft" combat system (Horssey and Fielding, 2017): first, identify an arguer's atti- tude roots and themes, and then choose a proto- typical rebuttal that aligned with those drivers instead of invalidating those. In this work, we are the first to explore Jiu-Jitsu argumentation for peer review by proposing the novel task of attitude and geedee rebuttal generation. To this end, we enrich an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. To facilitate this process, we recast established annotation concepts from the domain of peer reviews (e.g., aspects a review sentence is relat- ing to) and train domain-specifie models. We then propose strong rebuttal generation strate- gies, which we benchmark on our novel dataset for the task of end-to-end attitude and theme- guided rebuttal generation and two subtasks.
Reviewer
Attitude
Clarity
Overall (OAL)
Theme
Description,
The paper is neither nicely written nor easy to follow
Rebuttal Action
Reject Critictom
Canonical Rebuttaal
We Invite the referee to be speciflc about the sections of the original menuacript that read more dirification, allowing revise these sections.
Author
Figure I: Review Canonical Rebuttal Mapping via intermediate steps. The review has the attitude root 'Clarity' and theme 'Overall'. The canonical rebuttal is for the rebuttal action 'Reject Criticism'.
Recently, the analysis of review-rebuttal dynam- ics has received more and more attention in NLP research (Cheng et al., 2020; Bao et al., 2021; Ken- nard et al., 2022). For instance, previous efforts fo- cused on understanding the links between reviews and rebuttals (Cheng et al., 2020; Bao al., 2021) and on categorizing different rebuttal actions (c.g.. thanking the reviewers; Kennard et al., 2022). Gao et al. (2019) find that well-structured rebuttals can indeed induce positive score change. However, writing good rebuttals is challenging, especially for younger researchers and non-native speakers. For supporting them in writing effective rebuttals, argumentation technology can be leveraged.
1 Introduction
Peer review, one of the most challenging arenas of argumentation (Fromm al., 2020), crucial el- ement for ensuring high quality in science: authors present their findings the form of publication, and their peers argue why it should or should not be added to the accepted knowledge in a field. Of- ten, the reviews are also followed by an additional rebuttal phase. Here, the authors have chance convince the reviewers to raise their assessment scores with carefully designed counterarguments.
In computational argumentation, existing efforts towards generating counterarguments are based on surface-level reasoning (Wachsmuth et al., 2018; Alshomary et al., 2021; Alshomary and Wachsmuth, 2023). That is, existing research fo- cuses directly rebutting the opponents' claims based on what is evident from the argument's surface form. In contrast, researchers also pro- posed theories that explain how one's arguments are driven by underlying beliefs (Kiesel et al., 2022;
'Code at https: thwb. wtwp trr.iiituurgmmttttttiionn for_ooreerrttlllssss

---

Liscio et al., 2022). One such concept, rooted in psychology, relates to "attitude roots' and corre- sponding finer-grained 'attitude themes'. These attitude roots are the underlying beliefs that help sustain surface opinions (Hornsey and Fielding, 2017). Based on this idea, the authors further pro- posed 'ju-Jitsu argumentation' a rebuttal strategy that relies understanding and using one's at- titude roots to identify generic but customizable counterarguments, termed canonical rebuttals. As those are aligned with the opponent's underlying beliefs, Jiu-Jitsu argumentation promises to change opinions more effectively than simple surface-lewe counterargumentatioo. In this work, we acknowl- edge the potential of leveraging the latent drivers of arguments and are the first to explore Jiu-Jitsu argumentation for peer-review rebuttals.
canonical rebuttals. We hope that our efforts will fuel more rescarch on effective rebuttal generation.
2 Jiu-Jitsu Argumentation for Peer Review Rebuttals
We provide an introduction Jiu-Jitsu argumenta- tion and the task we propose.
2.1 Background: Jiu-Jitsu Argumentation
'Jiu-Jitsu' describes dloe-combat-basee fight- ing system practiced as form of Japanese mar- tial arts.? The term 'Jiu' refers to soft or gentle, whereas 'Jitsu' is related to combat or skill. The idea of Jiu-Jitsu is to use the opponent's 's strength to combat them rather than using one's own force. This concept serves as an analogy in psychology to describe argumentation style proposed for per- suading anti-scientists, c.g., climate-change skep- tics, by understanding their underlying beliefs ('at- titude roots', c.g., fears and phobias; Hornsey and Fielding, 2017). The idea is to use one's attitude roots to identify counterarguments which more ef- fectively can change one's 's opinion, termed canoni- cal rebuitals. A canonical rebuttal aligns with the underlying attitude roots and is congenial those. They serve as general-purpose counterarguments that can be used for any arguments in that particular attitude root-theme cluster.
Concretely, we propose the task of attitude root and theme- gunned rebuttal generation. In this con- text, we explore reusing established concepts from peer review analysis for cheaply obtaining reviewer attitude roots and themes (Kennard et al., 2022: Ghosal et al., 2022): reviewing aspects, and re- viewing targets (paper sections). We show ample Figure 1: the example review sentence criticizes the clarity of the paper without being spe- cific about its target. The argument thus driven by the attitude root Clarity and the attitude theme Overall. The combination of these two drivers can be mapped to an abstract and generic description of the reviewer's beliefs. Next, given specific rebut- tal action, here Reject Criticism, canonical rebut- tal sentence can be retrieved or generated, serving as template for further rebuttal refinement.
2.2 Attitude Root and Theme-Guided Peer Review Rebuttal Generation
We explore the concept of attitude roots for the domain of peer review. Our goal is to identify the underlying beliefs and opinions reviewers have while judging the standards of scientific papers (cf. $3). Note that good rebuttal in peer reviewing is also dependent the specific rebuttal action (Ken- nard et al., 2022) to be performed like mitigating criticism vs. thanking the reviewers. For example, in Figure I, we display the canonical rebuttal for 'Rejecl Criticism'. For the same attitude root and theme, the canonical rebuttal for the rebuttal action "Task Done would be "We have significantly im- proved the writing. re-done the bibliography and citing, and organized the most important theorems and definitions into a clearer presentation.". We thus define a canonical rebuttal as follows:
Contributions. Our contributions are three-fold: (1) we are the first to propose the novel task of attitude root and theme-guided peer review rebut- tal generation, inspired by Jiu-Jitsu argumentation. (2) Next, we present JITSUPEER, an enrichment to an existing collection of peer reviews with attitude roots, attitude themes, and canonical rebuttals. We build JITSUPEER by recasting established concepts from peer review analysis and training series of models, which we additionally specialize for the peer-reviewing domain via intermediate training. (3) Finally, we benchmark range of strong basc- lines for end-to-end attitude root and theme-guided peer review rebuttal generation as well for two related subtasks: generating an abstract review de- scriptions reflecting the underlying attitude, and scoring the suitability of sentences for serving as
Definition. 'A counterargument that is congenial to the underlying attitude roots while addressing
Phttps: http://enwwwwie wikipedia. artiiii..oo//iiiii org/wikiiJujuta..

---

them. It is general enough serve (as tem- plate) for many instances of the same (attitude root-theme) review uples while expressing specific rebuttal actions.'
Rebuttal Actions. Rebuttal sentences are directly annotated with the corresponding rebuttal actions. We show the labels in the Appendix (cf. Table 7).
(Ghosal et al., 2022). The second dataset we employ is benchmark re- source consisting of 1, 199 reviews from ICLR 2018 with 16, 976 review sentences. Like DIS- APERE, it comes with annotation layers out of which we use a single type, Paper Sections. These detail to which particular part a target paper a review sentence referring to (e.g.. method, prob- lem statement, etc.). We hypothesize that these could serve as attitude themes. Our intuition is that while our attitude roots already represent the underlying beliefs about work (e.g., comparison), the target sections add important thematic informa- tion to these values. For instance, while missing comparison within the related work might lead the reviewer to question the research gap, missing com- parison within the experiment points to missing baseline comparisons. We show the paper section labels in the Appendix (cf. Table 6).
SEER-RRVIEEWAAANA
Given this definition, we propose altitude root and theme- theme-guided rebuttal generation: given peer and rebuttal action the task is to generate the canonical rebuttal based on the attitude root and theme of rea.
review argument
3 JITSUPEER Dataset
We describe JITSUPEER, which allows for attitude and geome-guided rebuttal generation by linking at- titude roots and themes in peer reviews to canonical rebuttals based on particular rebuttal actions. To fa- cilitate the building of JITSUPEER, we draw on ex- isting corpora, and on established concepts, which we recast to attitude roots and attitude themes. We describe our selection of datasets (cf. $3.1) and then detail how employ those for building JrT- SUPEER (cf. $3.2). We discuss licensing informa- tion in the Appendix (cf. Table 5).
3.1 Starting Datasets
3.2 Enrichment
Our final goal is to obtain corpus in which view sentences are annotated with attitude roots and themes that, turn, link to canonical rebut- tal sentences given particular rebuttal actions. As detailed above. DISAPERE already provides us with review sentences, their attitude roots (i.e., re- view aspects), and links to corresponding rebuttal sentences annotated with actions. The next step is to further enrich DISAPERE. For this, we follow three-step procedure: (I) predict attitude themes using
DISAPERE (Kennard et al., 2022). The starting point for our work consists of reviews and corre- sponding rebuttals from the International Confer- ence Learning Representation (ICLR)² in 2019 and 2020. We reuse the review and rebuttal texts, which are split into individual sentences (9, 946 re- view sentences and 11, 103 rebuttal sentences), as well as three annotation layers:
Review Aspect and Polarity, Individual review sen- tences are annotated as per ACL review guidelines along with their polarity (positive, negative, neu- tral) (Kang et al., 2018; Chakraborty et al., 2020). We focus on negative review sentences, as these are the ones rebuttals need to respond to (2, 925 re- view sentences and 6, 620 rebuttal sentences). As attitude roots, explore the use of the reviewing aspects hypothesize that reviewing aspects represent the scientific values shared by the com- munity, e.g., papers need to be clear, authors need to compare their work, etc.
(2) describe at- titude root and theme clusters (automatically and manually), (3) identify a suitable canonical rebut- tal through pairwise annotation and ranking. The enrichment pipeline is outlined in Figure 2.
3.2.1 Theme Prediction
For finer- finererainedd typing of the attitude roots, we train models on PEER-REVIEW REVIEW-ANALLZE and pre- dict themes (i.e., paper sections) on our review sentences. We test general-purpose models as well as variants specialized to the peer review domain.
Ceview-Rebuute Links. Individual review sen- tences are linked rebuttal sentences that answer those review sentences. We use the links for re- trieving candidates for canonical rebuttals.
Models and Domain Specialization. We start with BERT (Devlin al., 2019) and RoBERTa (Liu et al., 2019), two popular general- pprppse models, as well as SciBERT (Beltagy et al., 2019), which is specifically adapted for scientific text. We compare
http:::://cdr.cc//offeeeeee///// https://iclr. httprrrcccccoooooceceeeeee/////e

---

DISAPERE
Enrichment Pipeline
Subetance
Paper Sections/ Thomes
Peer- Review-Analyes
Step Thema Prediction
DISAPERE
Ovenal (DAL)
Ciarty
The prosentation is not profesatonal,
Aspects/ Attitude Roots
Ithikk this is very internting direction, but present paper somewhat
Reviews
overall looks
subrisaion
Fine-tuning Language Models
Blank this is ittensating
nushed
(DML
direction, but the
preseett paper
somewhat
Step 2. Root- Theme Cluster Descriptions
nether wittien nor eaary follow
Human Acnottttion Extractive Summerizaiie
Canonical Rebuttal Identificatio
Rebuttals
Step
Rebuttal Actions
Reject
speetfic seciooe monusorist noad mere clariication allwwing us revire these seations.
We infite the referee be
Reject Criticlom
knvite refrree to be the origiine...
The reforee dhara contact
We tneile the refenme be specific about the sactions the original..
Fitteing
Preference Collection
Preference Ranking
Figure 2: Structure of our starting dataset, DISAPERE along with the Enrichment Pipeline used for the construc- tion of our dataset, JITSUPPER. The Enrichment Pipeline consists of three main steps that finally link review sentences from each attitude root-theme cluster to the canonical rebuttals for different rebuttal actions. The three main steps are in blue boxes and the sub-steps are in yellow boxes.
those variants, which further specialize our- selves the fine-grained domain of peer reviews using Masked Language Modeling (MLM) (Guru- rangan et al., 2020; Hung et al., 2022).
Model
Precision
Recall
Micro-F1
RANDOM
49.48
.001 09.87
000
16.14
.00
MAJORITY
33.24
.000 49.48
.0.
39.76
1000
BERT
71.92
010 57.05
63.63
RoBERTa
70.56 013 59.49 008
64.55
003
SciBERT
72.64 00066600066000000000000000 1.6.9221.00 64.45
.004
Configuration, Training, and Optimization. For domain specialization, we conduct intermediate MLM either on all DISAPERE reviews (suffix *ds "alaall) or on the negative sentences only (suffix *ds jis_nee'). We grid search over learning rates À E {1- 10-4,3 10-5) and batch sizes b {16, 32). We train for 10 epochs with early stopping (valida- tion loss, patience of 3 epochs). For the theme pre- diction, multi-label classification task, we place a sigmoid classification head on top the transform- ers and fine-tune on PEER-REVIE REVIEW- SEEVIEE--AAAYZEE ANALYZE (70/10/20 trin-validattonn-ess split) for a max- imum of 10 epochs. We grid search over different learning rates (A E (1 10-5, 2 10-5,3 10-5)) and use early stopping based the validation per- formance with a patience of 3 epochs.
BERT & my
72.03
.17 (0..01077.555 1.AT5=0000
64.35
7004
ROBERTA asses
71.03 + .00 59.52 + 000 64.77
.002
SciBERT mss
72.66 + .05 18.533::000
64,83
003
BERT RoBERTAs
72.38
I 1004 $7.36.1.0077 64,00
7002"
69.93
- 104 60.47 + .013 64.86 .005
SciBERT ac
68.89 + 000 60.61 + .008 64.49 .004
Table 1: Results for general puropae and domain- specialized models on theme enrichment task over 5 random runs. We report Precision, Recall, and Micro-Fl on the SERR-REVIEWAANEE the best result in bold. We underline the 4 best perform- ing models and separate model variants with dashes.
test and highlight
MAJORITY and RANDOM baseline.
Results and Final Prediction. All transformer models outperform the baseline models by huge margin (cf. Table 1). SciBERT neg yiclds the best score outperforming its non-specialized counter- part by more than percentage point. This points the effectiveness of our domain specialization, Ac- cordingly, we run the final theme prediction for Jrr- SU SUPEER with the fine-tuned SciBERT negn
Evaluation. On PEER-REVIEWE AEREW--NNLYZEE.C conduct 5 runs starting from different random seeds and report the average performance across those runs. We select the model with the best micro- Fl for the final prediction. We compare against a
neg- For en-

---

suring high quality of our data set, we only preserve predictions where the sigmoid-based confidence is higher than 70%. This way, we obtain 2, 333 re- view sentences annotated with attitude roots and attitude themes linked to 6, 614 rebuttal sentences. This corresponds to 143 clusters.
(relating 2, 332 review sentences) are candidates for canonical rebuttals, we reduce the size of our candidate set using scores obtained from two-types of suitability classifiers. Afterwards, we manually compare pairs of rebuttal sentences from the re- duced set of candidates. Finally, we compute ranks given the pair-wise scores and select the top-ranked candidate the canonical rebuttal.
3.2.2 Root-Theme Cluster Deseriptions
We add additional natural language descriptions to each attitude root-theme cluster. While these are not necessary for performing the final generation, they provide better human interpretability than just the label tuples. We compare summaries obtain automatically with manual descriptions.
Candidate Set Reduction. To reduce the set of canonical rebuttal candidates to annotate in later step, we obtain scores from two predictors: the first filter mechanism relies on confidence scores from binary classifier, which predicts rebuttal sen- tence's s overall suitability for serving as a canonical rebuttal and which we train ourselves. Second, the prototypical nature of canonical rebuttals plays an important role, we additionally use specificity scores from SPECIFICITELLER (Li and Nenkova, 2015). (1) For training our own classifier, we an- notate 500 randomly selected sentences for their suitability as canonical rebuttals. The annotator is given our definition and has to decide whether or not, independent from any reference review sen- tences, the rebuttal sentence could potentially serve as a canonical rebuttal. We then use this subset for training classifiers based on our general-purpor and domain-specific models with sigmoid heads. We evaluate the models with 5-fold cross-vaie validation (70/10/20 20 split). We train all models for 5 epochs, batch size 16, and grid search for the learning rates E [1 10 10-5, 2 10 10-5,3 5,3 10 10-93. The results are depicted in Table 3. As SciBERT neg achieves the highest scores, choose this model for filtering the candidates: we predict suitabilities on the full set of rebuttals and keep only those, for which the sigmoid- based confidence score is >95%.
Summary Generation. For the Human-written descriptions, we display < sentences randomly sampled from each review cluster to human la- beler. We then ask the annotator to write a short abstractive summary (one sentence, which summa- rizes the shared concerns mentioned in the cluster sentences). For this task, we ask a CS Ph.D. student who has experience in NLP and is familiar with the peer review process. For the automatic summaries, we simply extract the most representative review sentence for each cluster. To this end, we embed the sentences with our domain-specific SciBERT ds_ neg (averaging over the last layer representations). Fol- lowing Moradi and Samwald (2019), the sentence with the highest average cosine similarity with all other review sentences is the one extract.
Evaluation. Having a manual and an automatic summary for each cluster in place, we next decide which one to choose as our summary. We show the summaries together with the same S 5 cluster sentences an annotator and ask them to select the one, which better describes the cluster. We develop an annotation interface based on INCEpTION (Klie et al., 2018) and hire two additional CS Ph.D. stu- dents for the task.* 4 All instances are labeled by both annotators. We measure the inter-annotator agreement on the 99 instances and obtain a Cohen's 's k of 0.8. Afterwards, a third annotator with high expertise in NLP and peer reviewing resolves the remaining disagreements.
(2) SPECINCITELLER a pre-trained feature- based model, provides scores indicating whether sentences are generic or specific. We apply the model to our 500 annotated rebuttal sentences and observe that good candidates obtain scores between 0.02 0.78, indicating lower specificity. We thus use this range to further reduce the number pairs for rebuttal annotation. The complete filtering pro- cedure leaves us with 1, 8455 candidates.
3.2.3 Canonical Rebuttal Identification
Manual Annotation. For manually deciding on canonical rebuttals given the pre-filtred set of can- didates, we devise the following setup: we show a set of < review sentences from an attitude root
Last, we identify canonical rebuttals for each atti- tude root-theme cluster given particular rebuttal ac- tions. To this end, we follow a three-step procedure: first, theoretically all 6, 614 rebuttal sentences
$ As we will only use this classifier as a rough filter, we do not doubly-annotate this data.
"The annotation interface is presented in Appendix 7.4

---

(e.g.. 'Clarity") and attitude theme (e.g.. 'Overall")
Model
Mac-F1
Acc
RANDOM
46.9 0.05
56.4
cluster. We additionally pair this information with a particular rebuttal action (e.g.. 'Reject Criticism )
MAJORITY
71.3 0.04
83.2
BERT
93.6 0.06
97.0
Next, we retrieve two random rebuttal sentences that (a) are linked to any of the review sentences in that cluster, and (b) correspond to the rebuttal action selected. The annotators need to select the best rebuttal from this pair (again, interface imple- mented with INCEpTION), which is common setup for judging argument quality and ranking (e-g.. Habernal and Gurevych, 2016; Toledo al., 2019). For set of n rebuttal sentences available for a particular (attitude root, attitude theme, re- buttal action)-tuple, the pairwise labeling setup re- quires judgments for n(n - 12/2 pairs (in our case, 1, 102). We recruit 2 CS Ph.D. students for this task. In initial stage, pairs for two attitude roots (Clarity, Meaningful Comparison). We obtain Cohen's x of 0.45 (a mod- erate agreement, which considered decent for such highly subjective tasks (Kenmard et al., 2022)). We calculate MACE-based competencies (Hovy et al., 2013) and choose the annotator with the higher competence (0.82) to complete the annota- tions.
RoBERTa
97.0
SciBERT
96.0
BERT
94.7
96.0 0.03
RoBERTaaues
93.2
9.04
SciBERT
96.0 0.06
97.0 0.04
BERT
94.9
73.6
RoBERTag
94.4 = 0.05
96.2 + 0.04
SciBERT
96.0 = 0.06
97.0 + 0.06
Table 2: Performance of different models on the canon- ical rebuttal identification task in terms of Macro-Fi (Mac-F1 (1) and Accuracy (Acc) averaged over 5 folds.
800
500
let them doubly annotate
300-
1+200
100-
Substance
Soundress
Clarty
Replicabiltt
Originaity
Meaningful
Motivation
Other
Correctness
Comparison
Impaac
(a) Reviews
30-
25-
20-
15-
Canonical Rebuttal Selection. Following Haber- nal and Gurevych (2016), we obtain the best re- buttals from the collected preferences based on Annotation Graph Ranking. Concretely, we create a directed graph for each root-theme-action clus- ter with the rebuttal sentences as the nodes. The edge directions are based on the preferences: if A is preferred over B, create A + B. We then use PageRank (Page al., 1999) rank the nodes (with each edge having a weight of 0.5). The lowest-ranked nodes are the canonical rebuttals the node has very few or no incoming edges.
10-
0
Substance
Clarity
Soundres
Originality
Manningul
Replicability
Motivation
Other
Coneecness
Companson
Impact
Themes
Figure 3: Final data set analysis: number of review sentences and number of themes per attitude root.
soundness and utility of the methodology. In total, we identified 302 canonical rebuttals for different attitude roots and rebuttal actions.? Our canonical rebuttals can be mapped to 2, 219 review sentences (out of 2, 332). The highest number of canonical rebuttal sentences relate to the rebuttal action Task Done and the attitude root Substance. In Table 3, show examples of some of the canonical rebut- tals. We clearly note that different attitude root- theme deseriptions connect to different canonical rebuttals (e.g., Concede Criticism in Clarity and Substance).
3.3 Dataset Analysis
The final dataset consists of 332 review sen- tences, labeled with 8 attitude roots (aspects in AISAPE and 143 themes (paper sections in PEER SERR-REEIEWW-W REVIEW-ANALYZE) ANALYZE). We show label distri- butions in Figures! and 4, and provide more anal- ysis in the Appendix. Most review sentences have the attitude root Subsiance, which, also, has the highest number of themes (29). The most common theme is Methodology followed by Experiments and Related Work. This is intuitive since reviewers in machine learning are often concerned with the
"Ot of these, we obtained 117 without annotation since these were the only rehuttal sentences left after reducing the candidate set.
'The annotation interface is presented in Appendix 7.4

---

Atdtode Roots
Description
Canoaical Rebuttak
"We sigtifcuntly sihlogaphy oeganzeed importase theornne
Concade Critklaw:
indeed few
nicely DAT casy
puper phtaiing could Love bees beties, you pointing this out.
Coriy
into dleser procrccion.
Under Deseriptiona Metlad
Fursor Work: "Whiee we have proviial same
Reject Connoeer: 'As far as exglairing the mothod of combirstion, and the ussoisted nullenatio popprtics we lave tried to do this in detail section 1 (Apenaach)
diggunolic statiias, undeniandinn tiis actlaed deoply that will help fiel interating future resseck."
Atuwer:
"Whiee we put this expetimeut into the appendis japer
Incomplete details ta pe: peffomancc of the nethod
Strwvioring:
The experiment section badk more detiilal inalyyis
now, change soo much compared to the sulutined veoiin.
wlich cas intulively explain how well the propoeed methodd prrform on the bendumeris'
if the reviewere agrre wr world slan be very luppr inclule this experimnet the rasin paper."
Subetsncn
improvennne ovee beaclincs
York Done: "We provide detaile explasation the experiental wrinn funter esperinnaal catto-f-llessil pefformanee reqporos to The Common conoeres about experimenni setling and results:
Concod Crtiitin "Sevesleeees,
you meatingtul to emplasite cus inpervement over the stato- tninin mothods."
Table 3: Canonical rebuttals for different rebuttal actions (in italics), attitude roots, and theme descriptions.
# of canonical rebuttals
100
Models
r
MAP
MRR
RANDOM
0.010
13.86 - 0.01
13,90 0.01
80-
BERT
0.241
33.56 0.01
35.59
60-
RoBERTa
0.447
35.41 1.01
35.40 1.01
40-
SciBERT
0.470
36.73
36.72
BERT de nng
0.583*
39.20
39.13 70.02
20
RoBERTas my
0.454
36.21 0.01
36.21
Substance
Sourdress Correctness
Clarity
Originality
Repiiabiilty
Motivation
Impact
Meaninguul
Compaisson
Other
BERT I
SciBERT de neg
0.468
0.488
"38.31 0.03
36.67 0.01
48.3
36.68 0.01
RoBERTa all
0.323
37.82 0.03
37.67
0.03
SciBERT, alll
0.491
35.98
0.01
36.00 0.01
Figure 4: Final dataset analysis: number of canonical rehuttals per attitude root.
Table 4: Results of non-specialized and specialized vari- ants of BERT, RoBERTa, and SciBERT on the canonical rebuttal scoring Task. We report Pearson's Correlation Coefficient (r), Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) averaged over runs. (*) statistically significant differences (p < 0.05).
4 Baseline Experiments
Along with end-to-end canonical rebuttal genera- tion, we propose three novel tasks on our dataset.
train-validationn-tsst on the level of the number of attitude roots (4 - 1 ---3) 3). The total number instances amounts to 5941 out of which we use 2723-1450-1769 for the vain-validation-test. We experiment with all models described in $3.2. in fine-tuning setup. Following established work on argument scoring (e.g., Gretz et al., 2020; Holter- mann et al., 2022), we concatenate the description with the action a using a separator token (e.g., [SEP] for BERT). We grid search for the optimal number of epochs e E 21,2,3,4 3,3,44 3,4,5) 5} and learning rates À E (1 10-4, 2 10-4) with a batch size = 32 and select models based on their perfor- mance on the validation set. We also compare to baseline, where we randomly select score be- tween 0 and 1. We report Pearson's 's correlation coefficient (r), and, since the scores can be used for ranking, Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).
4.1 Canonical Rebuttal Scoring
Task Definition. Given natural language descrip- tion d, and rebuttal action a, the task is to predict, for all rebuttals r E R (relating to particular at- titude root-theme cluster), a score indicating the suitability of r for serving as the canonical rebuttal for that cluster.
Experimental Setup. The task amounts re- gression problem. We only consider combinations of rebuttal actions and attitude root-theme clus- ters that have a canonical rebuttal (50 attitude root- theme cluster descriptions with 3, 986 rebuttal sen- tences out of which 302 are canonical). We use the PageRank scores from before (cf. $3.2.3) our prediction target for model training. 8 To avoid any information leakage, we split the data into
"For canonical rebuttals obtained without pairwise anno- tation (these rebuttals were the only ones being predicted as canonical by our canonical rebuttal identifier and had no other candidates for comparison) we set the score to 0 (since the lower the score, the better the rebuttal)). Vice versa, for all other rebuttals, the score is set to 1.
Results. From Table 4, we observe that most of the domain-specializee models perform better than their non-specialized counterparts. SciBERT,

---

BART
20-
30- Pogasus
16-
BART
T5
10
Pegasus
10-
5-
Ts
2 Full
# of shots
# of shots
Figure ROUGE-1 variation on End2End Reuttal Generation task of BART, Pegasus, and TT with increasing number of shots.
Figure 5: ROUGE-1 variation on the Review Descrip- tion Generation task BART, Pegasus, and T5 with an increasing number of shots.
has the highest Pearson correlation across the board, however, BERT ds_gg neg has the highest rank- ing scores. The use of other cluster-related informa- tion such as representative review sentences, and paraphrasing the descriptions may lead to further gains which we leave for future investigation.
4.3 End2End Canonical Rebuttal Generation
Task Definition. Given review sentence rey, and a rebuttal action a, the task is generate the canonical rebuttal c.
Experimental Setup. We start from 2, 219 review sentences, which have canonical rebuttals for at least action. As input, we concatenate rev with placing a separator token in between resulting in 17, 873 unique review-rebuttal action instances, We use the same set of hyperparameters, models, and measures as before (ef. $4.2) and experiment with full fine-tuning, and zero-shot as well as few- shot prediction. For these experiments, we apply a 70/ 10/20 splits for obtaining train- vaaii--viiaiiion--iii portions on the level of the canonical rebuttals (302 rebuttals linked to 17, 873 unique instances).
4.2 Review Description Generation
Task Definition. Given a peer review sentence rev, the task is generate the abstract description d of the cluster to which rev belongs to.
Experimental Setup. Our data set consists of 2, 332 review sentences each belonging to one of 144 clusters with associated descriptions. We ap- ply vrin-validattin--ees split (70/10/20) and experiment with the following seq2seq models: BART (Lewis et al., 2020) (bart-large), Pega- sus (Zhang et al., 2020) (pegasus-large), and T5 (Raffel et al., 2020) (t5-large). We grid search over the number epochs e E {1, 2,, 5) learn- ing rates A E {1 10 5- 10-4, 1, 10-5) with a batch size b = 32. We use beam search with 5 beams as the decoding strategy. We run this ex- periment in full fine-tuning setup well as in zero- and few-shot scenarios (with random shot selection). We report the performance in terms of lexical overlap and semantic similarity (ROUGE- (R-1), ROUGE-2 (R-2), ROUGE-L (R-L) (Lin, 2004), and BERTscore (Zhang* et al., 2020)).
Results. The differences among the models are in-line with our findings from before (Figure 6, full results in Table 12): BART excels in the zero- shot and few-shot setups, and T5 starts from the lowest performance but quickly catches up with the other models. However, the models' performances grow even steeper than before, and seem to reach a plateau already after two shots. We think that this relates to the limited variety of canonical rebuttals and to the train-test split on the canonical rebuttal level we decided to make the task is to generate templates, and generalize over those. With seeing only few of those templates, the models quickly get the general gist, but are unable to generalize beyond what they have been shown. This finding leaves room for future research and points at the potential of data efficient approaches in this area.
Results. We show the R-1 scores in Figure (full results in Table 11). Interestingly, all mod- els exhibit very steep learning curve, roughly doubling their performance according to most mea- sures when seeing single example only. BART excels in the 0-shot and 1-shot setup, across all scores. However, when fully fine-tuning the mod- els, T5 performs best. We hypothesize that this relates to T5's larger capacity (406M params in BART vs. 770M params in T5).
Related Work
Peer Review Analysis. In recent years, research on peer reviews has gained attention in NLP, with most of the efforts concentrated on creating new datasets to facilitate future research. For instance, Hua et al. (2019) presented a data set for analyzing discourse
"We use the default voberta-large model for evaluation.

---

structures in peer reviews annotated with different review actions (c.g., EVALUATE, REQUEST). Simi- larly, Fromm et al. (2020) developed a dataset that models the stance of peer-review sentences in rela- tion to accept/reject decisions. Yuan et al. (2022) extended peer review labels with polarity labels and aspects based on the ACL review guidelines (as in (Chakraborty et al., 2020)). Newer works focused mainly linking the review sections to the exact parts of the target pupers they relate to (Ghosal et al., 2022; Kuznetsov al., 2022; Dy- cke et al., 2023). Overall, only few works focused on understanding the dynamics between review and rebuttal sentences. Exceptions to this are pro- vided by Cheng et al. (2020) and Bao et al. (2021) who study discourse based on sentence-wise links between review and rebuttal sentences. Kennard al. (2022) proposed a dataset that unifies existing review annotation labels and also studied review- rebuttal interaction.
publicly available. We believe that this dataset will serve as valuable resource to foster further re- search in computational argumentation for writing effective peer review rebuttals.
Limitations
In this work, we present a novel resource in the domain of peer reviews, JITSUPEER. Even though we develop our data set with the goal of foster- ing equality in science through helping junior re- searchers and non-nativn speakers with writing re- buttals in mind, naturally, this resource comes with number of limitations: JITSUPEER contains dif- ferent attitude roots and attitude themes along with the canonical rebuttals derived from the peer re- views from ICLR 2019 and 2020. ICLR top- tier Machine Learning Conference and thus the taxonomy developed in this process is specific to Machine Learning Community and does not cover the peer reviewing domain completely (e.g., natural sciences, arts, and humanities, etc). Thus, the source will have to be adapted as the domain varies. The canonical rebuttals also do not form closed since the original dataset DISAPERE, from which we started, does not contain rebuttals for ev- ery review sentence. Accordingly, the peer review to canonical rebuttal mapping is sparse. We there- fore, and for other reasons, highlight that writing rebuttals should be numan-in-the-loww process where models trained on JITSUPEER provide assistance by generating templates that be fur- ther refined for writing customized rebuttals.
Computational Argumentation and Attitude Roots. Computational argumentation covers the mining, assessment, generation and reasoning over natural language argumentation (Lauscher et al., 2022b). Here, most works focused on analyzing the explicated contents of an argument, e.g., w.r.t. its quality (Toledo et al., 2019; Gretz al., 2020), or structure (Morio et al., 2020), and on generat- ing arguments based on such surface-level reason- ing (Slonim et al., 2021). In contrast, Hornsey and Fielding (2017) analyze the underlying rea- sons driving peoples arguments. In similar vein, Lewandowsky et al. (2013) study the motivated rejection of science and demonstrate similar at- titudes toward climate-change skepticism. Fasce et al. (2023) extend this theory to the domain of anti-vaccination attitudes during the COVID-19 era. We borrow this idea and adapt it to the domain of peer reviewing to understand scientific attitudes.
Acknowledgements
This work has been funded by the German Re- search Foundation (DFG) part of the Research Training Group KRITIS No. GRK 2222. The work of Anne Lauscher funded under the Excellence Strategy of the German Federal Government and the Federal States.
6 Conclusion
this work, we explored Jiu-Jitsu argumentation for peer reviews, based on the idea that review- ers' comments are driven by their underlying atti- tudes. For enabling research in this area, we created JITSUPEER, novel data set consisting of review sentences linked to canonical rebuttals, which can serve as templates for writing effective peer review rebuttals. We proposed different NLP tasks on this dataset and benchmarked multiple baseline strate- gies. We make the annotations for JITSUPEER
We thank Aniket Pramanick, Nils Dycke, Luke Bates, and Haishuo Fang for their valuable feed- back and suggestions a draft of this paper. We would also like to extend our gratitude to Jing Yang, Tilman Beck, Cecilia Liu, Maral Dadvar, and In- derdeep Minhas for their help and support in anno- tating some portions of our dataset.

---

References
Angelo Fasce, Philipp Schmid, Dawn L Holford, Luke Bates, Iryna Gurevych, and Stephan Lewandowsky. 2023. A taxonomy of anti-vaccination arguments from a systematic literature review and text mod- elling. Nature Human Behaviour, (19):1462-1488))
Milad Alshomary, Shahbaz Syed, Arkajit Dhar, Martin Potthast, and Henning Wachsmuth, 2021. Counter- argument generation by attacking weak premises. In Findings of the Association for Computational Lin- guisties: ACL-IJCNLP 2021, pages 1816-1827, On- line. Association for Computational Linguistics.
Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddhartha Bhargava, Ruoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2020. Argument mining driven analysis of peer- reviews. In AAAI Conference on Artificial Intelli- gence.
Milad Alshomary and Henning Wachsmuth. 2023. Conclusion- based couttr--rguueen generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 957-967, Dubrovnik, Croatia. Associ- ation for Computational Linguistics.
Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao, 2019. Does my rebuttal matter? insights from major NLP conference. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Techmologies. Volume I (Long and Short Pupers), pages 1274-1290 1290, Min- neapolis, Minnesota. Association for Computational Linguistics.
Jianzhu Bao, Bin Liang. Jingyi Sun, Yice Zhang, Min Yang. and Ruifeng Xu. 2021. Argument pair extrac- tion with mutual guidance and inter-sentence relation graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3923-3934, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. PLOO ONE, 17(1):1-29.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SeiB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing TEMNLP-IJCNLEE pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguisties.
Shai Gretz, Roni Friedman. Edo Johen-Karlit As- saf Toledo. Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. Proceed- ings of the AAAI Conference on Artificial Intelligence. 78805):7805-7813.
Souvie Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analy- sis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, page 207-216, New York, NY, USA. Association for Computing Machinery.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342- 8360, Online. Association for Computational Linguistics.
Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 7000-7011, Online. Association for Computa- tional Linguistics.
Ivan Habernal and Iryna Gurevych. 2016. Which argu- ment is more convincing? analyzing and predicting convincingness of web arguments using bidirectional LSTM. In Proceedings of the 54th Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1589-15999 1599, Berlin, Germany. Association for Computational I Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedingy of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Carolin Holtermann, Anne Lauscher, and Simone Ponzetto. 2022. Fair and argumentative language modeling for computational argumentation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7841 7841-7861, Dublin, Ireland. Associa- tion for Computational Linguistics.
Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023, NLPeer: A unified resource for the computational study of peer review, In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049- 5073, Toronto, Canada. Association for Computa- tional Linguistics.
Matthew J Hornsey and Kelly S Fielding, 2017. Attitude roots and jiu jitsu persuasion: Understanding and overcoming the motivated rejection of science. The American psychologist, (255):49-477

---

Dirk Hovy, Taylor Berg- Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1 130, Atlanta, Georgia. Association for Computational Linguistics.
textual Model of Text-based Collaboration in Peer Review. Computational Linguistics, 48(4):949-99)
Anne Lauscher, Federico Bianchi, Samuel R. Bowman, and Dirk Hovy. 2022a. SocioProbe: What, when, and where language models learn about sociodemo- graphics. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 7901-7918, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 2131-22 1131-2137, Minneapolis, Minnesota. Association for Computational Linguistics.
Anne Lauscher, Henning Wachsmuth, Iryna Gurevych, and Goran Glavaš. 2022b. Scientia Potentia Est- On the Role of Knowledge in Computational Argumen- tation. Transactions of the Association for Computa- tional Linguistics, 10:1 1091--12222
Stephan Lewandowsky, Gilles E. Gignac, and Klaus Oberauer. 2013. The role of conspiracist ideation and worldviews in predicting rejection of science. PLOS ONE, 8 8(10):1-11.
Chia-Chien Hung. Anne Lauscher, Simone Ponzetto, and Goran Glavaš. 2022. DS-TOD: Efficient domain specialization for task-oriented dialog. In Findings of the Association for Computational Linguistics: ACL 2022. pages 891-904, Dublin, Ireland. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising requence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguistics.
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 1647-1661, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Junyi Li and Ani Nenkova. 2015. Fast and accurate prediction of sentence specificity. Proceedings of the AAAI Conference on Artificial Intelligence, 29)1..
Neha Kennard, Tim O'Gorman, Rajarshi Das, Ak- shay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and An- drew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1234-1244 1234-1249,,66 Seattle, United States. Association for Computational Linguistics.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona. Spain. Association for Computational Linguistics.
Enrico Liscio, Alin Dondera, Andrei Geadau, Catholijn Jonker, and Pradeep Murukannaiah. 2022. Cross- domain classification of moral values. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2727-2745, Seattle, United States. Association for Computational Linguistics.
Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind argu- ments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 4459-4471, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv preprint.
Milad Moradi and Matthias Samwald. 2019, Clustering of deep contextualized representations for summa- rization of biomedical texts. ArXiv preprint.
Jan- ChrrCiiittpph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEPTION platform: fachine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
Gaku Morio, Hiroaki Ozaki, Terufumi Morishita, Yuta Koreeda, and Kohsuke Yanai. 2020. Towards bet- ter non-tree argument mining: Proposition-Ieved bi- affine parsing with task-specific parameterization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3259- 3266, Online. Association for Computational Lin- guistics.
Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and Resubmit: An Inter-

---

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking Bringing order to the web. In The Web Conference.
Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLJ) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel Bowman. 2021. When you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1112-1125, Online, Association for Computational Linguisties.
Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Karlik,, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Friedman- Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Ja- covi, Charles Jochim. Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger. Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liber- man, Yosi Mass, Amir Menezel, Shachar Mirkin, Guy Moshkowich, Shila Koif-an, Matan Or- bach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, Ilya Shnayderman, Aya Soffer, Artem Spector, Benjamin Sznajder, As- saf Toledo, Orith Toledo-Ronel Elad Venezian, and Ranit Aharonov. 2021. An autonomous dehating system. Nature, 591(7850)33793884
Ian Tenney, Dipanjan Das. and Ellie Pavlick. 2019. BERT rediscovers the classical NIP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguistics.
Assaf Toledo, Shai Gretz, Edo Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Auto- matic argument quality assessment new datasets and methods. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5625-5635, Hong Kong, China. Association for Computational Linguistics.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge.
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 241-20 1, Melbourne, Australia. Association for Computational Linguistics.

---

Dataset
License
% of
Datasets
Review Labels
DISAPERE
Creative
Com-
sents
mons
Attribution-
Soundness- Soundness-Correctees Correctness
22.15
NonCommercial
4.0
Clarity
21.26
DISAPERE
International
Public
Suhstance
30.52
License
Originality
7.24
PEER REVIEW-ANALYZE JITSUPEER
MIT License
Replicability
7.48
Creative
Com-
Motivation- Impact
3.15
mons
Attribution-
Meaningful-CCopariioo
5.81
NonCommercial 4.0
Other
2.39
International
Public
Methodolegy
27.73
License
Introduction
2.84
PEER REVIEE ANALYZE
Related Work
5,05
Table License details the different datasets used in
Problem Definition
3,43
Datasets
1.49
the paper.
Experiments
4.72
Results
3.51
Tables Figures
2.22
7 Appendix
Analysis
0,90
Future Work
0,43
7.1
Information on dataset licenses
Overall
6.82
Bibliography
0,99
The licenses for different datasets used in the paper are listed in Table 5.
External
1.41
Table 6: Distribution of review sentences in DIS- APERE and SEER-RREIEE--AALSS
7.2 Dataset details
For PEER- REVIEW-ANALYZ these labels constitute 60. 60.24% of review sentences, the rest of the review sentences are annotated with different combinations of these labels.
We list the distribution of review sentences (in %) for the datasets used in our work namely, DIS- Table We show the distribution rebuttal sentences with respect to rebuttal actions in Table 7. We display part of the label hierarchy developed for JITSU- PEER in Table 8. We plot the number of canonical rebuttals with respect different rebuttal actions in Fig 9. "Task Done and 'Answer' the most com- mon rebuttal actions for the annotated canonical rebuttals.
APERE and SEAAR-REEEEWAAAYYY
probing, we follow (Lauscher et al., 2022a), and use a batch size of 32 with learning rate, A= 1e-3 using the Adam optimizer. The models are trained for 100 epochs with early stopping the valida- tion set with patience of 5. The results aver- aged over runs. We use the standard train/dev/test splits from DISAPERE.
From Table 9, we observe that the best- performing models are pre-trained RoBERTa, SciB- ERT, SciBERT neg. and SciBERT
7.3 Probing of the Domain Specialized Models
order to gauge the quality of review represen- tations of the different transformer models (pre- trained and domain specialized), we additionally perform probing task. We first start with describ- ing the probing task. We adopt classic prob- ing procedure following previous works (Tenney al., 2019; Zhang et al., 2021: Lauscher al., 2022a) where classifier placed on top of the frozen features. Following (Zhang et al., 2021; Lauscher et al., 2022a), we use a simple two-layer feed-forward neural network on top of the frozen features. For given text, we average over the rep- resentation of all the tokens except the special to- kens. We perform the probing task on DISAPER: We the already available classification tasks in the dataset such as 'Aspect', 'Review Action', and "Fine-grained Review Action' classification as the proxy tasks. We evaluate the models in terms of Accuracy (Acc) and Macro-F1 (F1) metrics. For
7.4 Annotator Details and Interface Design
We recruited 2 CS Ph.D. students for the different tasks that require human annotation while creating dataset, JITSUPEER. Initially, experimented with varying degrees of expertise but found that CS Masters students struggled with the task, which is why we resorted to recruiting PhD students. We trained the annotators initially for hour to explain the guidelines and then answered their questions when needed. We adapted the INCEpTION plat- form (Klie et al., 2018) for carrying out annota- tions. In the dataset creation pipeline, there tasks that explicitly require human intervention: i) Root-Theme Cluster Description (cf. $3.2.2) and, ii) Manual Evaluation for Canonical Rebuttal Iden- tification (cf. $3.2.3). The interfaces for both these tasks are presented in Fig 7 and Fig respec-

---

INCEPTION
Aperagett
Innditon
Iriissr
Goup tylle
Sottyacoee
Racoi reneeinii
lewaver todk authon reed better expan netiaato ffweaoo OWWheer wasing prrvied theegerratto setir pu
autttruun poae
Fasoe taiiing of catonar wertng nodel vas botter cntterecc pracice upp Rolottoon, tha grd sowwe, bu andt.
Given that the espect of this cluster of reviews asp wwtpmmiiviiioommmpettttt and Paper section EXP, which the following description describes cluster
maiy
expattmnts stucnts images bitrryer nod partinaaal
weplenngulllaaoooooo of soliee of ircezvedrati rathed
Figure 7: Snapshot of the interface used for collecting annotations for the Theme Cluster Description Task. The attitude root here is 'Motivation Impact' (asp motivatioviionn iiniioonnmmat)) and the theme "Experiment (EXP). Gitdy
ICEPTION B Foeeds Duuuoeer)
& Xilewck th 0ch
Libe
aaterma tepes unuudder stelemonds
The saparis ninly whitie crcathr eay
To trugetio i accot, this prpppe rewd somn chanes i writer munn, mambe freedll tonswormn wiach varlye ipparattdd becrrurssyy
stigeal paper prepraatio.
The wringg nest sats lattre
commoris warsirs the taa muriniee
prodenne)
Given aspect this cluster reviews asp_ clarity and Paper OAL, following rebuttals selected as canonical rebuttal for the rebuttal action 'rebuttal
We hove Aicil cartly Inprovd witl redore the Bie oelyy dina loga sbod noot tep Bearemoad defictins ato dester prosentatoon We sttarmabed to mack impoos carty of adding torter apprrsoonnnnn and connccing oper place.
Figure 8: Snapshot of the interface used for preference collection for the Manual Evaluation for Canonical Rebuttal Identification Task. The attitude root 'Clarity' cisp_clarity) and the theme 'Overall (OAL) with respect the rebuttal action, 'rebuttal done'.
Rebuttal Actions
% of rebuttal sents
from the set of rebuttals R that corresponds to the attitude them- cluster.
Answer
33.93
Reject criticism
15.57
Experimental Setup. We formulate this problem as a ranking task. The number of cluster descrip- tions and rebuttals is the same as in the Canonical Rebuttal Scoring Task (cf $4.1). We implement BM25 based information-retriovy system as a base- line for this task. We also use different bi-encoder- based models built using S-BERT (Reimers and Gurevych, 2019). As in Task 1, we split the data on the level of attitude roots (1 1 3) for training- validation-test. We concatenate d and using [SEP] token. We encode the concatenated clus- ter description and rebuttals independently and use cosine similarity as a distance measure for training and inference. We finetune pre-trained sentence transformer models on this task along with train- ing new sentence transformer models using em- beddings from different language models (BERT, RoBERTa, SciBERT, and their domain-specializze variants). We perform grid search over learning rates E (le-5, 5, 2e 2). epochs e (1, 2, 4, 5) and batch sizes, b (16, 32). We report Mean Av- erage Precision (MAP) and Mean Reciprocal Rank (MRR).
Structuring
15.79
The task done
9.70
Concede criticism
3.69
The task will be done
2.11
Accept future work
1.28
Accept praise
0.05
Mitigate criticism
3.31
Reject request
1.13
Refute question
1.27
Contradict assertion
0.76
Summary
9.86
Social
1.07
Followup questions
0.32
Other
0.16
Table 7: Distribution of rebuttal sentences (in %) for different rebuttal actions DISAPERE
tively.
7.5 Canonical Rebuttal Ranking Task
Task Definition. As another variant of the Canon- ical Rebuttal Scoring Task (cf $4.1), now seek to directly identify the canonical rebuttal using re- trieval: given a description and a rebuttal action o, the task is to retrieve the canonical rebuttal c

---

Aftiude roots
Themes
Descriptions
Exumppe review sents.
The paper 1s not nicely writenn or naber casy to follow
The paper can beneht from proofreoding..
Ovenal
"I think this very interesting divection, is somewhat unclear"
present paper
Clanny
Whhtt di Algorithm I deseription ?"
Unclear description of method
Melhod
"The method very con fusingly presented and roquirts both knowledge of HAT well more than one reoding understind"
"Simply bocause continous vuriables similar expesiments
Nox enough novelty
have been reported before'.
Experiments
experiments
"Also, find experiments dase vaction and simillar
(seens similar to presious woek)
previous works and even the conclusions are similar.
Originality
"As the authors admii, the main resuli
especially surpeising..
Not cnough originality in results
Roslts
"although there must some small innovations, thought that all
results had more or loss been proven by Dupuis and co anthoes:"
Attocking CRBMs highly relevanz and should included as biseline!.
Rolated Woek
Missing Basclines
'The paper should also talk about the denils of ARNet and discuss
Meuningful Comparison
differenes,
they
reluted work'
More comparisons
don't t think thss is zeally farr comparison;
necoeed with vartations the proposed method
woold have liked to have resulss for the unmodified rewand function.
Results
"The human score 91.4% based majority vose.
which should be compured with an ensemble of deep lsaming prediction.
4.3, there
giamutee tlat the
Claim on the datisets is questionable
Dainset
intersection between the training set and test set empoy..
Soundness Convetnnss
'The should used before the best compression scheme selected.'
Incomplete ablation of tubless and ligures
"The unceriainties produced by CDN in Figure 2 soems strange..
Tables and Figures
"It didd also nor mutch sumbers in Talk of the appendis."
Thersfore, may good idea
suthors analyze
Analysis
Lack of analysis
the correlstion belween FSM changes and sccuraty clanges!
Substance
qualitaive analysis how modulation actuiily by systems"
'BSD 300 only contans 300 images, and it would be good
Less mumber datasers used
Taisset
more diverse of imnges considered""
"Thind, the datasets used this paper are nother limited'
Limited insight based on design choices
"The main drawhack of the paper is that seems to be more engineering
insight somi supervised inglnestng-fncused fncused.
Methodology
and doesn't provide
Motivation Impuct
'Thus we may only apply the prepesed model on a few tasks with esaully known E* The resuits overall not very impressive..
Limited impact of resulis
Rosalts
"The resalis of the paper do not give major insights into what are the preforred icchniques for training GANSs and certainly why and under what cincumetances they 'Il work""
Missing implorenntaiin details of telated work as basellines
"The gragh neursl setwoeks mead in model
deserihed in the paper.
only reference Paliwal et al (2019) is given.
Related Woek
would helpful have parggaph deseribing this architsccture, for readers not familiar with the referenced puper."
Replicability
'Are the authors willing releuse the code? Overall the model looks complacated and the appondis not sufficient to reproduce the result the puper',
Missing detaills regrodacibility of results
Rosats
'I trast that the authors did fact achieve these results but cannot figure out how why.
I Chink paper does hit conference.
Reasons for rejection (not Gi for conference, contains several weaknesses, etc.)
is hoteer to be presented in Demonstration section a *ACL conference"",
Ovenil
Overall, fed
piper falls
penmises,
accepeance at this time.
Other
The above are not ciued this papee'.
Bibliography
Missing citations
"The cited paper 'Learning an adapeive leanning rate schodlle" does not appear online."
Table 8: Attitude roots along with themes, descriptions, and example review sentences in our proposed dataset,
JITSUPEER. The attitude roots are the same as the aspects in DISAPERE (Kennard et al., 2022) and the themes related to the paper sections proposed in PEER- REEE--REEEEE--LLLEE (Ghosal et al., 2022).

---

Review Action
Fine Review Action
Aspect Identifcction
Model
Acc
F1
Acc
F1
Acc
F1
Acc
F1
BERT
78.1 1 0.008
47.4
0.012
37.0
0.010
37.2 = 0.030
=7.200020
54.2
46.1
RoRERTa
78.7 + 0.008
75.8 0.009
45.4 0.002
363 00009
42. 0.010
31.0-0.001
55.4
47.7
SciBERT
+ 0..0000.004
76.7 0.005
45.2 0.020
=
41.5 0,010
29.5 -9.500.020
58.7
47.2
BERT&
76.8
69.3 0.007
0.020
33.6 0.040
"1. 0.000
0.020
50.8
4223
ROBERTRUA
0.10
0.051
4..4
39.6
46.3
a oe.....
0.034
0.022
48.6
39.5 0,020
00010
55.5
48.3
BERTaall
0.01
72.7 + 0.040
0.010
38.6 0.010
32.9 0.040
24.3 0.020
53.3
43.2
RoDERTIe at
SciBERTa
79.7 + 0.050
76.4 004+000000
45.8
36.0 = 0.050
41.7 0.010
31 0.010
55.7
48.3
Table 9: Results for experiments on the first phase of the theme prediction task Probing. We demonstrate results for three tasks Review Action, Fine Review Action, and Aspect Identification in terms of Accuracy (Acc) and FI. Bold indicates the best-performing models across all the tasks on average.
Models
MRR
MAP
BM25
0.209 10.011
0.211 1 0.015
allmini-Im
0.218 0.016
0.219 1 0.002
paraphrase-alber'
0.195
0.196 +
BERT
0.227 - 0.003
0.229 1 0.002
RoBERTa
0.226 = 0.001
0.227 + 0.001
SciBERT
0.209 - 0.001
0.209 1 0.002
BERT nos
0.231 = 0.003
0.232 + 0.002
RoBERTas
0.220 - 0.002
0.221 1 0.002
SciBERT AL dumss
0.220 0.002
0,222 + 0.002
BERT
0.229 0.077
0.230 + 0.012
RoBERTas SciBERT
0.193 0.018
0.195 - 0.012
0.193 0.017
0.195 + 0.012
Table 10: Results on Canonical Rebuttal Ranking Task. We report Mean Average Precision (MAP), Mean Reciprocal Rank (MRR) for this task. ds_ dsg_ ds_all represent models that are domain specialized on all the reviews and negative reviews from DISAPERE respectively. (*) represents pre- trained sentence transformer models obtained from [https: h/wpp:////wwww.e sbert t. net/index. html] and finetuned for this task.
Results. Table 10 shows the performance for different models. Interestingly. BM25 has competitive performance with different finetuned models which in line with Kennard al. (2022). However, the best model is the finetuned sentence transformer model for BERT trained on negative reviews (BERT ds neg). This in line with our findings for Canonical Rebuttal Scoring Task (see Table 4) where BERT neg had the best ranking scores.
shots Models
R-1
R-2
R-I.
BERTsc
BART
10.75
2.43
9.31
0.16
Pegasus
10.28
2.30
8.96
0.16
TS
5.29
0.93
4.87
-0.05
BART
21.49
5.70
19.87
0.359
Pegasus
17.26
6.06
16.58
0.314
TS
16.79
4.21
15.94
0.260
BART
19.11
5.26
18.42
0.342
Pegasus
17.94
6.55
17.33
0.322
TS
18.36
6.14
17.50
0.276
7.6
Review Description Generation Task
BART
42.36
33.64
42.17
0.496
Full
Pegasus
45.73
37.83
46.36
0.516
The full results for this task provided in Ta- ble 11.
TS
46.47
37.85
45.95
0.520
Table 11: Performance of BART, Pegasus, and T5 on the Review Description Generation task. We report the zero-shot, few-shot, and full fine-tuming performance in terms of ROUGE-I (R-I), ROUGE-2 (R-2), ROUGE-L. (R-L) and BERTscore.
7.7
End2End Canonical Rebuttal Generation Task
The full results for this task are provided in Ta- ble 12.

---

SnSwEr
siructuring
ortiiism
reect
critiim
ready
summary
socil
futume
miligtte
criicism
refute
question
followup
contradict
2ocertion
other
of canonical rebuttals
Figure 9: Final dataset analysis: number canonical rebuttals per rebuttal action.
shots Models
R-1
R-2
R-L.
BERTscore
BART
I1.53
0.87
9.20
0.005
0
Pegasus
10.97
0.16
2.73
-0.32
T5
3.08
0.79
8.88
-0.002
BART
13,43
11.01
0.132
Pegasus
20.64
16.48
0.249
TS
19.93
15.09
0.203
BART
22.04
3.42
16.90
0.256
2
Pegasus
23.26
4.94
18.18
0.270
TS
20.23
3.59
16.27
0.224
BART
22.14
3.88
15.67
0.218
Full
Pegasus
21.52
3.87
15.69
0.193
T5
21.58
3.86
15.98
0.195
Table 12: Performance of BART, Pegasus, and T5 on the End2End Canonical Rebuttal Generation task. We report the zero-shot, few-shot, and full-fineste full-finetuinng per- formance in terms of ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L) and BERTscore.


Reviews:
[{"review_id": "ALUyXu4ph1", "reviewer": "Reviewer_5iUR", "summary": "", "questions": "Question A: Most of the human annotations are done by one or two CS PhD students. It would benefit to provide more details of annotators, for instance, were these annotators same as the authors of the paper? If not, what was the criteria used to for recruitment besides familiarity with peer-review process? How much interaction between authors and the annotators was possible and how did it impact annotation quality? \n\nSuggestion:\nA discussion about who is intended as the user of this proposed system and what is the user expected to benefit from such a system could help strengthen the paper further.  For instance, as much as I understand, with the current framing of the paper, the intended user of the system is the author of a paper under review and suggestions of canonical rebuttals are supposed to provide ideas on how to construct a review. The system is not expected to output exact rebuttals that can be used directly, but most relevant rebuttal patterns that would require customization. I think that is a very reasonable approach and a discussion of such system use can benefit readers. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comprehensive feedback and for appreciating our work! \n\n# **Questions**\n#### $Q1$. **[Annotators] “Most of the human annotations are done by one or two CS PhD students. It would benefit to provide more details of annotators, for instance, were these annotators same as the authors of the paper? If not, what was the criteria used to for recruitment besides familiarity with peer-review process? How much interaction between authors and the annotators was possible and how did it impact annotation quality?”**\n\n$Ans$.  We thank the reviewer for their question and agree that more details on the selection of the annotators will enrich the discussion. Initially, we experimented with varying degrees of expertise but found that CS Masters students struggled with the task, which is why we resorted to recruiting PhD students. The final set of annotators consists of two Ph.D. students (one of whom is also author of this paper). We trained the annotators initially for 1 hour to explain the guidelines and then answered their questions when needed. We will provide a more detailed description in case of acceptance of our work.\n\n# **Suggestions**\n#### $S1$. **[UseCase] “A discussion about who is intended as the user of this proposed system and what is the user expected to benefit from such a system could help strengthen the paper further. For instance, as much as I understand, with the current framing of the paper, the intended user of the system is the author of a paper under review and suggestions of canonical rebuttals are supposed to provide ideas on how to construct a review. The system is not expected to output exact rebuttals that can be used directly, but most relevant rebuttal patterns that would require customization. I think that is a very reasonable approach and a discussion of such system use can benefit readers.”**\n$Ans$. We appreciate your suggestion on the use case and, in case of acceptance, we will add this discussion to our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task is novel (attitude and theme prediction followed by canonical rebuttal identification) and well motivated. \n2. The authors provide a resource which can be used by other researchers to advance computational modeling for peer-review rebuttal generation. \n3. The authors provide extensive experiments and the paper is well written. ", "reasons_to_reject": "I don't think i have any concerns. ", "questions_to_authors": "Question A: Most of the human annotations are done by one or two CS PhD students. It would benefit to provide more details of annotators, for instance, were these annotators same as the authors of the paper? If not, what was the criteria used to for recruitment besides familiarity with peer-review process? How much interaction between authors and the annotators was possible and how did it impact annotation quality? \n\nSuggestion:\nA discussion about who is intended as the user of this proposed system and what is the user expected to benefit from such a system could help strengthen the paper further.  For instance, as much as I understand, with the current framing of the paper, the intended user of the system is the author of a paper under review and suggestions of canonical rebuttals are supposed to provide ideas on how to construct a review. The system is not expected to output exact rebuttals that can be used directly, but most relevant rebuttal patterns that would require customization. I think that is a very reasonable approach and a discussion of such system use can benefit readers. ", "paper_topic_and_main_contributions": "The authors propose the task of peer-review rebuttal generation using attitude roots and themes, motivated by an argumentation style inspired by the Jiu-Jitsu combat strategy. The key contributions are \n1. new task of attitude and theme-guided rebuttal generation\n2. enrichment of an existing peer-review dataset with attitude roots, attitude themes, and canonical rebuttals. The authors use intermediate classification models trained on another related dataset to make annotation predictions. \n3. provide baseline methods for end to end task of attitude and theme-guided rebuttal generation. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": " ", "Typos_Grammar_Style_and_Presentation_Improvements": " "}, "extra_text": {}}, {"review_id": "Yxc6lepyjI", "reviewer": "Reviewer_7PTc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your feedback and appreciating our work! We address the concerns below.\n\n# Reasons\n\n#### $R1$. **\"No significant shortcomings have been identified. However, the token adjacent to the title should be removed to align with the conventions of academic writing.\"**\n$Ans$. We will incorporate your suggestion and update the title to “Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttal”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study presents a compelling argument, supported by a comprehensive evaluation.", "reasons_to_reject": "No significant shortcomings have been identified. However, the token adjacent to the title should be removed to align with the conventions of academic writing.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel task regarding attitude and theme-guided rebuttal generation. To achieve the goal, the authors adapt the established concepts from peer review analysis and develop domain-specific models.\nThe contribution of this paper includes the provision of a dataset and the conduct of a thorough evaluation. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ahEc1dzTdX", "reviewer": "Reviewer_b1L5", "summary": "", "questions": "Question A: In Section 3.2.1 Theme Prediction and Section 3.2.3 Canonical Rebuttal Identification, why don't you choose a more powerful model such as T5 or GPT for classification? Since the data consturction depends on the performance of classification, I wonder if a more powerful would be benificial. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback on our work. We justify the concerns below:\n\n# Reasons\n#### $R1$. **[Annotators] ”The number of annotators used for human annotation are quite small (one or two as described in the paper), which may results in the human annotation less convincing”**\n$Ans$. We thank the reviewer for sharing this important concern. However, in our case, we do not believe that having two annotators will result in lower data set quality. On the contrary: since we deal with a task in the domain of peer review, our annotations require scientific expert knowledge leading to a smaller pool of qualified annotators. Our annotator selection and our final agreement is comparable to other studies in this domain, pointing to the reliability of our dataset. For instance, Kuznetsov et. al (2022) choose two annotators and report an inter-annotator agreement (IAA) of $0.53$ (comparable to $0.45$ in our case), and Kennard et. al (2022) report an IAA range of $0.447$ - $0.605$ (lower than $0.45$ in our case).\n\n#### $R2$. **[Model Biases] “The dataset construction depends on the model performance of classification at middle steps, which may result in potential biases.”**\n$Ans$.  We thank the reviewer for their comment on the potential side-effects of supporting the data set creation with machine learning models – we acknowledge that, generally, models will always reflect the biases of their training data. This is exactly why we only keep the models’ most confident predictions through uncertainty thresholding (cf. ll. 290), or only to reduce the huge search space (cf. ll. 344) for the final manually conducted annotations. The latter is a standard technique in the literature (e.g., Simpson et. al (2018), Kuznetsov et. al (2022)) and we thus strongly believe in the high quality of our data set.\n\n#### $R3$. **[Annotation details] “Some details of the human annotation are missing. For example, the authors mention that \"We develop an annotation interface based on INCEpTION (Klie et al., 2018)\", but the questions for the annotation are not given.”** \n$Ans$.  We thank the reviewer for pointing out this missing detail. We will add the annotation guidelines along with the snapshots of the interface in the camera-ready version.\n\n# Questions\n#### $Q1$. **[More Powerful models] “In Section 3.2.1 Theme Prediction and Section 3.2.3 Canonical Rebuttal Identification, why don't you choose a more powerful model such as T5 or GPT for classification? Since the data consturction depends on the performance of classification, I wonder if a more powerful would be benificial.”**\n\n$Ans$. We agree with the reviewer that T5 and GPT are go-to candidates for generative tasks, and thus, we employed those for Review Description Generation (cf. ll 492) and End2End Canonical Rebuttal Generation (cf. ll 524). However, for the discriminative tasks, we focus on state-of-the-art models for scientific text analysis (Kennard et. al, (2022), Ghoshal et. al, (2022)) in-line with our goal of benchmarking strong baselines for future research. Accordingly, we test SciBERT, SPECTER, and their base variants BERT and RoBERTa (as a “more advanced BERT”). Additionally, we experiment with even finer-grained domain specialisation (for peer review texts), which we demonstrate to be effective. \n\n# Typos\n#### $T1$. **[Typo] “Typo in Line 367”**\n$Ans$. We thank the reviewer for catching this typo which we will correct in future versions of this manuscript.\n\n### **References**\nSimpson et. al, (2018): “Finding Convincing Arguments Using Scalable Bayesian Preference Learning” (Simpson et. al, TACL 2018)\n\nKuznetsov et. al, (2022): “Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review” (Kuznetsov et. al, Computational Linguistics 2022)\n\nKennard et. al, (2022): “DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions” (Kennard et. al, NAACL 2022)\n\nGhoshal et. al, (2022): “Peer review analyze: A novel benchmark resource for computational analysis of peer reviews” (Ghoshal et. al, PLOS One 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work is the first one to explore Jiu-Jitsu argumentation for peer review by proposing the novel task of attitude and theme-guided rebuttal generation.\n\n2. This paper construction a new dataset to for the task of end-to-end attitude and theme guided rebuttal generation and two subtasks.\n\n3. The paper also provides some benchmarks for the attitude and theme-guided rebuttal generation tasks.", "reasons_to_reject": "1. The number of annotators used for human annotation are quite small (one or two as described in the paper), which may results in the human annotation less convincing.\n\n2. The dataset construction depends on the model performance of classification at middle steps, which may result in potential biases.\n\n3. Some details of the human annotation are missing. For example, the authors mention that \"We develop an annotation interface based on INCEpTION (Klie et al., 2018)\", but the questions for the annotation are not given.", "questions_to_authors": "Question A: In Section 3.2.1 Theme Prediction and Section 3.2.3 Canonical Rebuttal Identification, why don't you choose a more powerful model such as T5 or GPT for classification? Since the data consturction depends on the performance of classification, I wonder if a more powerful would be benificial. ", "paper_topic_and_main_contributions": "This paper explores the problem of writing effective rebuttals in peer reviews. The authors propose the use of Jiu-Jitsu argumentation, inspired by the Jiu-Jitsu combat system, as a strategy for generating rebuttals that align with an opponent's underlying beliefs. The main contributions of this paper include:\n\n1) the introduction of attitude roots and attitude themes in argumentation;\n2) the enrichment of an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals;\n3) the exploration of the potential of leveraging attitude roots and themes in peer review rebuttals. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 367, \"Table 3\" should be modified as \"Table 2\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the novel task of attitude and theme-guided rebuttal generation based on argumentation is interesting and the proposed approach is sound. They highlighted some drawbacks in their reviews like the limited human evaluation, the lack of details about it, and the lack of deeper discussion about the potential biases on the dependence of the dataset on the model performance. The reviewers appreciated the author rebuttal which clarified most of the issues.
Decision: Accept-Main