Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation
Authors: Jiayu Lin
Rong Ye
Meng Han
Qi Zhang
Ruofei Lai
Xinyu Zhang
Zhao Cao
Xuanjing Huang
zhongyu wei
Venue: EMNLP 2023 Main (2023)
Keywords: Counter-Argument Generation
ArgTersely
Arg-LLaMA
Arg-Judge

Abstract: Counter-argument generation—a captivating area in computational linguistics—seeks to craft statements that offer opposing views. While most research has ventured into paragraph-level generation, sentence-level counter-argument generation beckons with its unique constraints and brevity-focused challenges. Furthermore, the diverse nature of counter-arguments poses challenges for evaluating model performance solely based on n-gram-based metrics. In this paper, we present the ArgTersely benchmark for sentence-level counter-argument generation, drawing from a manually annotated dataset from the ChangeMyView debate forum. We also propose Arg-LlaMA for generating high-quality counter-argument. For better evaluation, we trained a BERT-based evaluator Arg-Judge with human preference data. We conducted comparative experiments involving various baselines such as LlaMA, Alpaca, GPT-3, and others. The results show the competitiveness of our proposed framework and evaluator in counter-argument generation tasks. Code and data are available at https://github.com/amazingljy1206/ArgTersely.

Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation
Jiayu Lin', Rong Ye Meng Han', Qi Zhang',
Ruofei Lai Xinyu Zhang', Zhao Cao*, Xuanjing Huang', Zhongyu Wei"*
Fudan University, "ByteDance, 'Huawei Poisson Lab jiayulin22en fudan. edu. cn, wen..eeerraabbeedddneeeee.. com
xjzannh zywei)efudan, edu. cn
ha caozhao waaaaaoohaol@huueeeeaee )ehuawei com
Abstract
Tepic Can firearm restrictions reduce gun crime?
Counter- arguumnt generation a captivating area in computational linguistics seeks to craft statements that offer opposing views. While most research has ventured into paragraph-level generation, sentence-level counter-argument generation beckons with its unique constraints and brevity-focused chal- lenges. Furthermore, the diverse nature of counter-arguments poses challenges for eval- uating model performance solely based on n- gram-based metrics. In this paper, we present the ArgTersely benchmark for sentence-leve counter-argument generation, drawing from manually annotated dataset from the Change- View dehate forum'. We also propose Arg- LlaMA for generating high-quality counter- argument. For better evaluation, trained a BERT-based evaluator Arg-Judge with hu- man preference data. We conducted com- parative experiments involving various base- lines such as LIaMA, Alpaca, GPT-3, and oth- The results show the competitiveness of our proposed framework and evaluator in counter-argument generation tasks. Code and data are available https://gg thub. com/ amazingl jyl 2rrrl////////rrrreeeel
8
The jovensment ahoul strister regulations firearm ownership reduce related crimes.
Original Argutemn
While underssed the concenn relicing pun-related crim- consiōer tie balance between individual ii- wiety Stricter regllations fireamm ship may indinse uoo Scoond Aiscidars ristis lose absing muzzn mn hunder ther abiiry to prstect themoelves. Puttheroce, Eocusing solely on fream regulaaioos ts may ove- lonk aéfresin the underbing factors that contribuee to vissenes, sach menual health isturs and saclosononic dis- comprehensive sppronch that inchiles improved braltheaae, responsible gun swnership prugams, education effective cing gun-rvlated crimes while respecting individkal rights.
8
Panagraph-irnal
Strieter firearm megllatios line the rights Inu-iniding effectively oudressing crimes, menoal youurs and inclorconeric dispartiio.
8 Sontence
Figure I: An example that elucidates the difference between paragraph-level and sentence-leve counter- arguments. Topie words reflecting the discussion points are in bold. Words that are underlined and in the same color denote the key points shared between two counter- arguments.
sentence. It requires identifying the key points and formulating counter-argument in limited space. An example of the difference between paragraph- level and sentence-level counter- arguumet genera- tion shown in Figure 1.
1 Introduction
Counter-argument generation task aims to auto- matically generate statement that has differ- ent stance from the original argument (Toulmin, 2003; Damer, 2009). Existing works describe it paragraph-level generation task (Hua and Wang, 2018: Alshomary al.. 2021; Alshomary and Wachsmuth, 2023). However, sentence-level counter-argument generation can be quite different. The main challenge of sentence-leve generation is to condense the counter-argument into a concise
To address this challenge, we propose benchmark ArgTersely for sentence-level counter- argument generation. The dataset is derived from ChangeMy View (CMV), an online debate forum, and has been annotated by humans.
Recently, large language models, such as Ope- nAI ChatGPT and GPT-4 (Bubeck et al., 2023), PaLM (Chowdhery al., 2023), and LlaMAs (Tou- vron al., 2023a,b) have achieved great suc- cess and demonstrated remarkable performance in text generation tasks. By leveraging the pre- trained language model, we propose a framework,
"Conresponding author.
Phttps://www reddit, comrr//cangegggm/iie//

---

Arg-LlaMA, to generate high-quality counter- arguments. Our framework is pipeline com- prising (1) an instruction component, (2) a lan- guage model, and (3) a filter component. The instruction component comprises multiple Chain- of of-Thought (CoT; Wei et al., 2022) instructions addressing common errors in debates along with their corresponding reasoning steps. As for the lan- guage model, utilize instruct-tuning (Wei et al., 2021) on LlaMA-7b (Touvron et al., 2023a) with the Low-rank Adaptation (Hu et al., 2021) method. During inference, we employ multiple CoT instruc- tions as input for the language model and utilize the filter component to select the best candidate counter-argument as the output of the system.
2 ArgTersely
2.1 Task Formulation
The task input consists of two components: a topic and an original argument. (1) The topic, denoted as explains the premise of the dialogue and the focus of the debate.
(2) The original argument, denoted as æ, is a sentence containing the initial perspective stance put forward. The objective of this task is to generate sentence, y, that provides a coherent rebuttal response to x based on the given topic T.
2.2 Dataset Creation
We based our dataset annotation on the CMV dataset (Tan et al., 2016), sourced from the Change- VyView (CMV) subreddit CMV users post var- ious topics, with unique quoting dynamic: User quotes a segment of User A's statement (usually a sentence) and responds (often with multiple sen- tences). We extracted 20, 626 triplets from CMV, emphasizing reply relationships. Each triplet in- cludes a topic, a quoted statement (original argu- ment), and its reply. The reply, split into sentences, forms candidate set. Annotators select sentences from this set that counter the original argument, creating counter-argument pairs. Sentences are skipped if they are incomplete, lack a viewpoint, or breach our ethical guidelines (Appendix F). Any sentence flagged by an annotator for violating these guidelines is excluded. Our annotation process comprises data preprocessing, trial annotation, and formal annotation ensure dataset quality.
Previous work typically employed n-gram-based metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) for rapidly evaluating the qual- ity of counter-argument generation (Alshomary al., 2021; Schiller al., 2021). However, believe that these metrics do not effectively assess whether the generated sentences are pertinent and in line with human preferences. To this end, we propose incorporating model- metrics, Arg- Judge, as supplementary evaluation approach. Specifically, we trained BERT-based (Devlin et al., 2019) model, using the human preference data generated during the annotation process. In addition, introduce metric, ChatGPT Eval, which we obtain by using ChatGPT score the sen- tence's position and argument completion. More- over, we have made the human evaluation more specific by asking human annotators to assess the outputs based on five dimensions, which enables a comprehensive evaluation of the model's 's perfor- mance.
Data Preprocessing We segment User B's replies into candidate sentences using punctuation and re- move those with hyperlinks, emails, phone num- bers, emojis, etc. Grammar issues, such as capital- ization and spacing, are corrected.
Our contributions are mainly as follows:
Trial Annotation Before formal annotation, anno- tators are trained on rules and the annotation sys- tem. After that, they undertake trial annotation on 50 triplets. We retained annotators who displayed high consistency with our reference annotations. totaling 24.
We propose benchmark, ArgTersely, for sentence-level counter-argument generation and dataset annotated by humans.
We propose counter-argument generation framework, Arg-LlaMA. The framework is capable of generating high-quality counter- arguments.
Formal Annotation To minimize bias, we employ cross cnssssstnotation strategy. Two annotators assess the same triplet. and any discrepancies are resolved by third. Approximately 30% of the dataset is generated through arbitration.
We propose a novel, lightweight evaluator, Arg-Judge, which enables it to reflect the real ranking and is highly consistent with human evaluation.

---

Output: all birds fly. Penguin
Filtering(Arg-Juuee
Score: 0.8
Not all birds can fly. Penguin can not fly.
Oue bird's flght prove all birds can fly.
Ignoring flightiess hirds skews the argument.
Linear Transformation & Normalization
Pooling
Language Model
BERT-L
Focusing flying pigeun.
Penguins can't fl.
equivalent to "all birds".
BERT-Layer 1
Language Model
CLS
Inpux
CoT Instruction
CoTT Instruction
CoT Instruction
Candidate Sentence
Input:
All birds can fly because saw pigeon flying yesterday.
Figure 2: The overview of proposed framework Arg-LlaMA. First, CoT instructions guide the language model to identify errors. Next, the LM generates candidate sentences based on those errors. Finally, BERT-based filter selects the best counter-argument by scoring concatenated original argument and candidate sentence.
All annotators have substantial debate experi- ence and at least bachelor's degree. The anno- tation process spanned 42 days, yielding 31, 197 aggument-counterargumen pairs, each associated with the relevant topic. We highlight the ethical considerations during the annotation process, in- cluding potential risks, identifiable information, compensation, and annotation biases in Section 9, The statistics of Arg Tersely are shown in Table
lecting high-quality counter- argument. We employ LoRA and instruct-tuning methods to obtain an LM. Additionally, we leverage human preference data to train the filter.
During inference, we use CoT instructions as inputs the LM. After obtaining a series outputs from the LM, the filter will select the best counter- argument as output. The generation pipeline is detailed in Section 3.3.
Train
Valid
Test
3.1
Instruct-tuning the LlaMA
# topic
7911
2000
Instruction Set Creation In line with the self- instruct (Wang et al., 2023) approach, we initially generated 148 instructions based on 10 seed structions. Following a manual verification pro- cess, these instructions were expanded to form an Argumentation Instruction Set consisting of 2,772 instructions. Specifically, our specific implemen- tation differs from the self-instrue method in the following aspects:
# of pair
28197
1000
2000
Avg.
# words argument
21.74
21.57
19.96
words per counter argument counter-argument
25.09
27.44
34.92
Table The statistics Arg Terselyy
3 Arg-LlaMA
Figure shows the framework we proposed, Arg- LlaMA. It mainly composed of two parts: 1) a language model (LM) with instruct-tuning, for generating aunnterarruumet,, and 2) a filter, for se-
Our seed instructions focus on argument-related instructions, such as "Provide evidence sup- port the conclusion". "Point out its logical er- ror", etc. A detailed list of specific instructions

---

is shown in Appendix A and attached files.
employed is cross-entropy loss:
2. We use the ChatGPT 3 to generate instructions, enabling to generate more diverse and elabo- rate contexts.
[ = - log E (1,2,3,4) (1)
3.3 Generation Pipeline
Low-Rank Tuning Using the above Argumen- tation Instruction Set and Alpaca instruction set (Taori et al., 2023), we fine-tuned LlaMA-7b model with LoRA method. LoRA maps the weight update of the self-attention module projection ma- trix in the Transformer (Vaswani et al., 2017) ar- chitecture to a lower dimension and then returns to the normal output dimension. In our work, we performed LoRA on all Query/Key/Valee/OOppe projection matrices in the self-attention module.
The generation pipeline consists of three steps: I) provide CoT instructions guide the LM, 2) use the LM, generates outputs based on instructions, and 3) apply filtering to refine and obtain the final result by selecting the most appropriate counter- argument.
CoT Instruction Our generation pipeline starts with series of CoT instructions. We propose CoT instructions to guide the model in generating alistic and logical arguments through multi-step reasoning. Based on Kee's s (2006) debating theory, we design few-shot and multi-step reasoning tem- plates for several common errors debate. We roughly divide common errors into the following categories:
3.2 Training the Filter
The filter component is also a language model. We designed this component with the purpose of select- ing high-quality counter-arguments from candidate sentences.
Factual Error: a mistake in the presentation of fact.
Ranking Data for training Our training data, named Ranking Data (RD), originates from human preference data generated during the annotation process of ArgTersy dataset. Given an original argument z, assign ranking scores to candidate sentences based the following rules:
Logical Fallacy: errors in reasoning that un- dermine the validity of an argument.
Confirmation Bias: errors in selectively in- terpreting information in way that supports existing hypotheses.
Sentences selected by annotators that can form strong rebuttal relationship with x.
The specific formats of these instructions are listed in Appendix C. During inference, pro- vide the LM with set of instructions that corre- spond the aforementioned errors for generating the counter-arguments.
Sentences selected by the annotator but belonging to the same conversation as æ.
Safe reply, randomly selected from a prede- fined list, as listed in Appendix B
LM The LM serves two roles. Firstly, it acts as an error identifier, tasked with identifying errors within the original argument. Secondly, it generates candidate counter-argument for each instruction provided.
Sentences sampled from other conversa- tions.
We finally got 20,000 training samples and 800 testing samples, each sample consists of an orig- inal argument and four candidates. We denoted the original argument as the candidates list as Y [vI, 1/2 Va. val. and the ranking score for yı as s1, (1,2, 1).
Filter After the LM with various CoT instructions. get set of candidate counter-argument Y [/1 .i., y/n. The purpose of the filter is select the one that maximizes the probability as the output of the system:
Training Task The training task is learning to rank the candidates in the correct sequence. In this task. we assign the ranking scores of four candidates as the ground truth, with higher scores indicating lower quality.
y arg max magmax 1/x, v.), 1,... (2)
To optimize the parameters 0 of the filter, first used the parameters of BERT-base (Devlin al., 2019) initialize it. The loss function we
4 Experiments
4.1 Experiment Setup
WWo use gpt-3.5 turbo as in https://platform openai. wttp:/cccoooaaaaeeeeeeeee //tpss//ppt--//pp 3-5, and ditto.
Dataset Our experiments are performed on the test set of ArgTersely dataset. consists of 2,000

---

BLEU
ROUGE
METEOR
ChatGPT Eval
Arg-Judge
Word
# Param (b)
BART
5.20
11.75
6.38
30.40
7.17
11
0.1
GPT-2
13.52
15.74
14.51
31.47
23.26
15
0.8
DialoGPT
16,54
18.72
17.50
16.96
9.98
15
0.8
LiaMA
15.85
22.07
21.77
35.02
37.46
AlpA
15.56
18.67
17.53
46.82
47.51
36
7
GPT-3
10.42
13.97
12.66
45.85
42.33
15
175
Arg-LJaMA
18.60
22.41
19.29
50.48
55.78
38
7
Table 2: Main result on sentence-level counter-argumen generation. We report BLEU-1 (BLEU), ROUGE- L (ROUGE), METEOR, ChatGPT Eval, Arg-Judge, average number of words per sentence(# Word) and the number of parameters in each model(# Param). The best results are in bold. Our proposed model perform well in most metrics (Wilcoxon signed rank test (Kotz and Johnson, 2012), < 0.05).
triplets in the format of <topic, original argument, counter-argument>>
Automatic Evaluation First, we do not entirely disregard n-gram- based automatic evaluation met- rics that commonly utilized, including BLEU (Pa- pineni et al., 2002), ROUGE (Lin, 2004) and ME- TEOR (Lavie and Agarwal, 2007).
Implementation Details When training the hase LM with instruct-tuning, we use LlaMA-7b as the base model. We set the learning rate to 3 x 10-4, batch size to 256, gradient accumulation step to 16, and train the model epochs RTX3090 GPUs. The a and of the LoRA method are both 16. When training the filter, we use BERT-base as the base model. We set the learning rate to x 10-5, batch size to 64, and train on an NVIDIA RTX3090 GPU for 2 epochs. For training both models, we employed AdamW (Loshchilov and Hutter, 2018) as optimizer.
However, of greater importance, we present two model-based evaluation metrics to assess perfor- mance differences among different systems. A de- tailed explanation follows:
NVIDIA
ChatGPT Eval: We utilize two instructions to guide ChatGPT in generating the stance score (Sat) and the argument completeness score (Scom), both of which range from 0 to 100, The instructions we employed are outlined in Appendix D. The stance score assesses whether the original sentence and the generated sentence have opposing stances, while the completeness score gauges the generated counter-argument caliber, specifically if it makes logical sense. We employ a weighted average of these two scores to get the final score of ChatGPT Eval.
Models for Comparision We compare our system with several baselines:
BART (Lewis et al., 2020): pre-trained lan- guage model with encoder-dccoder structure, and we fine-tuned to adapt this task.
GPT-2 (Radford et al., 2019): pre-trained language model with decoder-only structure. DialoGPT (Zhang al., 2020): decoder-only language model which was trained on online dialogue corpus.
Sgpt = AS + -X)Scom
(3)
where À is set to 0.5 in our experiments. To reduce the uncertainty of ChatGPT provided, we set the temperature factor to 0.1.
LlaMA (Touvron al., 2023a): collection of models trained publicly available datasets, and we use LlaMA-7b.
Arg-Judge: order to ascertain the degree of relevance and informativeness of the generated counter-arguments, we adopt a "reverse valida- tion" approach using the Filter model that was trained in Section 3.2. To this end, we estab- lish Aug-Judge as the metric for evaluating the efficacy of this approach in identifying mean- ingful couuuer-arguments that
AlRA (Taori et al., 2023): model obtained by LoRA-tuning LlaMA-7b based on the Alpaca instruction set.
GPT-3 (Brown et al., 2020): large language model without instruct-tuning.
mere non- sense safe replies. Specifically, we normalize the average-pone pooled hidden before the softmax layer of the filter model 0 to get a continuous
4.2 Evaluation Metrics
Our evaluation metries include automatic evalua- tion metrics and human evaluation.

---

predicted score á E [0, 4]. We empirically de- fine the Arg-Judge score as
@Alpaca-LoRA
Grommaticality
Arg LLMA
8
9
Say max
8}
(4)
36'
We selected the hyper-parameter setting based observation that large language mod- els (such as Alpaca-LoRA and GPT-3) tend to generate sentences with scores concentrated be- tween and 0.8. Arg-Judge can thus enhance the distinguishability for these high-scoring sen- tences, while still maintaining the monotonicity, Human Evaluation Based on the work of Hua et al. (2019), we conducted a more detailed human evaluation. Five human judges are asked to rate arguments on Likert scale of (worst) to (best) across 5 dimensions to evaluate the performance of the systems:
Persuasiveness
Appropriateness
Logic
Content Richness
Figure 3: Result of human evaluation on grammaticality. appropriateness, content richness, logic and persuasive- ness.
also complemented with each other. For in- stance, BART and Dialog-GPT models tend generate stance-correct but logic-and-content- lacking "safe replies", which may receive ac- ceptable scores in BLEU, ROUGE, or ChatGPT Eval, but low scores from Arg-Judge. This ad- dresses the limitations of traditional metrics used in the past.
Grammaticality: assess whether output ad- heres to the rules of grammar.
Appropriateness: focus on whether the out- put is contextually suitable.
Content Richness: reflect the depth of infor- mation provided by output.
Human Evaluation We report the result of human evaluation in Figure 3 and Table 3, and we have the following observations:
Logic: measure the rationality of output.
Persuasiveness: show the extent to which readers are persuaded by output.
Our system outperforms in multiple dimensions and Top-1 metric. shows that the counter- arguments generated by our framework are more in line with human preference.
Additionally, We use Top-1 to represent the pro- portion of the best output. We emphasize the im- portance of human evaluation as it provides results that more aligned with human value, compared to automatic metrics based n- grgamm models.
Result of Human evaluation consistent with ChatGPT Eval and Arg-Judge we report in Ta- ble 2, which corresponds to our hypothesis that Im-based evaluator may be suitable for counter- argument generation.
4.3 Main Results
Automatic Evaluation As the results listed in Table 2, have the following conclusions:
Compared with other models, our system excels in appropriateness, logic, and persuasiveness This achievement can be attributed to the CoT instructions, that effectively guide the language model through multi-step reasoning.
Our model achieves the best results on most of the metrics. The components in our framework help our system to generate correct and fluent counter-argument.
Arg-LlaMA and Alpaca-LoRA outperform non- instruct-tuning models like LlaMA and GPT- 3. Because non-instruct-tuning models is tend to enhance and extend the original argument, rather than forming rebuttal relationship with it, which indicates stance error.
Top-1
Arg-LlaMA
62 %
Alpaca-LoRA
27%
LlaMA
10%
Comparing n-grams to reference sentences for argument generation tasks is often insufficient. Our proposed model-based metrics, ChatGPT Eval, and Arg-Judge, not only demonstrate con- sistency with the n-gram metric, but they are
Table 3: Result of human evaluation. Top-1 means the proportion of system output ranked as the first. The counter-arguments generated by our proposed model were rated by human judges to be of higher quality.

---

4.4 Ablation Study
ChatGPT
Arg- Judge
Models
Eval
We perform ablation studies to explore the role of different components in both training and gener- ation process. We explored four variants in addi- tion to the overall framework: 1) Instead of the argumentation instruction set, instruction set to instruct-tuning the LM. 2) We replace the LM with LlaMA-7b, which has not been fine-tuned by instructions. 3) We remove CoT instructions components and use a series of simple instructions, such as "Give me the counter- argument". 4) We remove the filter components and select output from the candidates randomly.
Arg-LlaMA
50.48
55.78
Training
argumentation instruction instruct-tuning
44.93
51.30
39.12
43.77
the Alpaca
Generation
chain-of- of-tuhhhtt multi-errors filter
39.64
48.13
36.27
51.90
42.79
35.47
w/ random select
43.45
39.06
Table 4: Ablation analysis includes both training and generation modules. ChatGPT Eval and Arg-Judge are both applied.
The results of the ablation study are presented in Table 4. We have the following findings:
5.1 Datasets for Validation
Using argumentation instructions during training is very helpful for the model. This clearly demonstrates the effectiveness of our pro- posed argumentative instruction set. We make the argumentation instruction set publicly acces- sible to benefit the wider community.
Ranking Data (RD): We use the test set of this dataset to check if the Arg-Judge can reflect the real ranking. As mentioned in Section 3.2, it has 800 testing samples. A sample includes original argument and four candidate counter-arguments.
Quality Selection Dataset (QSD): This dataset is used to check whether the Arg-Judge aligned with human evaluation. consists of 500 triplets in the format of coriginal argument, better counter- argument, worse counter-argument>> Given an original arguments from Arg Aersesy,yy we first used the ChatGPT to generate two counter-arguments and then manually selected as the better counter-argument and another the worse counter- argument.
Instruct-tuning matters. Simply generating from LlaMA affects performance, while instruc- tion tuning can help the model better adapt to argumentation scenarios, respond to the instruc- tions, and reason out correct rebuttals from CoT instructions.
Compared with common instructions, CoT instructions can produce higher-quality arguter-rgummens.. This is because CoT in- structions can give a logical chain and a multi- step reasoning process, which improves the qual- ity of output.
5.2 Validation Tasks and Comparisions
Multiple error templates can improve the quality of generated counter-arguments. Mul- tiple error templates can help the LMs discover potential errors from multiple perspectives, thus generating richer candidate sentences.
RD: Can Arg-Judge reflect the real ranking? Given the original argument, the task on the RD dataset is to select the best counter-arguman from four candidates. We use precision at one (P@) 1) to measure the ability of Arg-Judge reflect real ranking.
Filter component plays a crucial role in our system. It enables us to select high-quality argu- ments from candidate sentences, while random selection fail to achieve similar performance.
QSD: Is Arg-Judge consistent with human evalu- ation? Given the original argument, the task on the QSD dataset is to select a better counter-argument from two candidates. We use accuracy to reflect the consistency between Arg-Judge and human evalua- tion.
5 Validation of Arg-Judge Metric
order to explore the capability of our proposed Arg-Judge to reflect the actual ranking level and its consistency with human evaluation, we designed two corresponding tasks.
Comparisions We use BERT-base and ChaiGPT comparisons. adapt ChatGPT to these tasks, we constructed two instructions. Specific information about the instructions is in Appendix E.

---

5.3 Validation Result
Topic: How should the struggles of white upper middle-class individuals be addressed in the USA! Original Argument: The struggles faced by white upper middle-class individuals in the USA raises questions about the prospects and viability of in- vesting in the white working- let alone pro- viding support for communities of color in inner cities.
Arg-Judge can reflect the real ranking. The re- sult in Table 5 shows that the performance of Arg- Judge is better than ChatGPT and BERT. It means that Arg-Judge demonstrates sorting capabilities that reflect real-world scenarios after training. Arg-Judge is highly consistent with human eval- uation. Result is in Table 6. Based on the result, the consistency with human evaluation ranks in the following order from high to low: Arg-Judge, ChatGPT, and BERT. It shows the high consistency between Arg-Judge and human evaluation.
Ours: However, the struggles faced by white upper middle-class individuals in the USA should not be used as an excuse for not investing in the white working-class.
GPT-3: The white working-clast is not mono- lithic group. The white working-class is not a ho- mogeneous group. The white working-class is not a static group.
P@1
BERT 0.38
Alpaca-LoRA: The struggles faced by white upper middle-class individuals in the USA raises ques- tions about the prospects and viability of investing the white working-class and providing support for communities of color in inner cities.
ChatGPT 0.53
Arg-Judge 0.62
Table 5: Validation result on the RD dataset. It demon- strates the reliability of sorting the results of Arg-Judge score.
LlaMA: Let alone providing support for commu- nities of color in inner citics.
Accuracy
Table 7: Case study of an instance in the test set.
BERT
0.55
proposed CMV dataset, including the citation re- lationship between original posts and their cor- responding replies. Bolton et al. (2020) intro- duced Kialo, a dataset for sentence-level argu- ment stance classification and counter-argument generation. ArgTersely distinguishes itself the first aanon-and dataset of its kind with ranking data reflecting human preferences. Early work (Hua and Wang, 2018; Hua et al., 2019) fo- cus on how to introduce external knowledge into the system; Alshomary et al. (2021) developed system to identify weak points in arguments; Schiller et al. (2021) developed controlled argu- ment generation system, which is able generate arguments based on given information; Alshomary and Wachsmuth (2023) completed it through multi- task and multi-step reasoning. Our work primarily introduces benchmark and evaluation metrics for sentence-level argument generation. Methodologi- cally, we just establish usable baseline using LLM without introducing too much external knowledge.
ChatGPT 0.77
Arg-Judge 0.79
Table 6: Validation result on the QSD dataset. Arg- Judge shows high consistency with human evaluation.
6
Case Study
We illustrate the advantage of our model through a case study in Table 7. Alpaca-LoRA and LlaMA fail to adequately tap into the subtext of the original argument that the white upper middle class can- not help the working class when they are fighting each other. And without instruct-tuning, GPT-3 and LlaMA mostly consist of extensions additions the original text, lacking a compelling rebuttal. Whereas, our model recognizes this implicit logic and implements a counter-argument that it is not a plausible excuse.
7 Related Work
Counter-Argument Generation Datasets (Ji al., 2021; Yuan et al., 2021; Hua and Wang, 2018; Stab al., 2018) for counter-argument gen- eration mainly establish the rebuttal relationship in the conversation using automatic methods such as citation or reply detection. Tan et al. (2016)
Instrunttttt A series of recent works (Zhou et al., 2022; Ye et al., 2022; Singh al., 2022; Hon- ovich et al., 2023) generate instructions of task given a few examples. Singh et al. (2022) use LLM and reranking algorithm to generate human- interpretable instruction, which matches or even im-

---

proves upon human-written instruction. Honovich al. (2023) introduce the instruction induction challenge task and discover the ability to gener- ate instructions emerge when a language model is large enough. Wang et al. (2023) provide an almost unnotation-free method for aligning pre-trained lan- guage models with instructions. The overall pro- cess is an iterative bootstrapping algorithm, which starts off with limited seed set of manually writ- ten instructions that are used guide the overall generation. We fine-tuned the Arg-LlaMA model using the self-instruct approach, where we included seed instruction for a variety of related tasks of the counter-argument generation.
Ethical Considerations
Since we propose a new dataset ArgTersely, we solve some possible ethical issues this section. Potential Risk Our dataset is sourced from Change- MyView (CMV), a subcommunity on Reddit. Users must adhere to community rules", including restrictions on hate speech. We also formulate ethical guideline and require annotators to follow it We train annotators to mark and skip sentence violating the ethical guideline. Annotators were informed about potential risks. Our annotation process respects intellectual property and privacy rights.
Identifiable Information Our data is sourced from open platforms, safeguarding privacy. We also re- moved sensitive information such emails, phone numbers, and usernames during data preprocess- ing.
Adaptation As the size of the language model increases, the cost of fine-tuning also increases. A series of works (Shin et al., 2020; Li and Liang. 2021; Houlsby al., 2019) have studied various methods like prompt-tuning and adapter-tuning alleviate this problem. However, it's difficult to directly optimize the prompt, and introducing the adapter layer will cause a delay in reasoning. Considering that, Hu et al. (2021) proposed Low- Rank Adaptation (LoRA), which greatly reduces the number of trainable parameters for downstream tasks. Additionally, there have been advancements in extending Low-Rank Adaptation. Dettmers et al. (2023) proposed QLoRA that fine-tunes large mod- els on limited memory GPUs through 4-bit quanti- zation and Low-Rank Adapters. Chen et al. (2023) introduced LongLora that leverages sparse local attention and achieves context extension with mini- mal computation. Since our work does not involve long contexts in generation and does not prioritize optimization techniques, just utilize LoRA to fine-tune our model for efficiency.
Compensation We employed 24 part-time annota- tors, compensating them at $0.25 per conversation (equivalent to at least $3.75 per hour, with a cap of 2 hours per day), which surpasses the local mini- mum wage.
Annotation Bias We perform series of meth- ods to reduce the bias during annotation, includ- ing annotator training, trial annotation, and cross- annotation.
Limitations
While the experimental results demonstrate the ef- feetiveness of Arg-Judge, it important to note that our exploration of the consistency between human evaluation and language model evaluators (includ- ing ChatGPT Eval and Arg-Judge) was limited to specific set of scenarios. Furthermore, due com- putational resource constraints, we were unable train a larger-scale language model as an evaluator. Moving forward, our future research will involve expanding the evaluation of the language model evaluator across a broader range of scenarios and utilizing a larger-scale language model to enhance its capabilities.
8 Conclusions
In this paper, we introduce benchmark ArgTersely for sentence-level counter-argument generation. Specifically, we present human-annotated dataset and develop a language model based on argumenta- tion instructions. We further construct a framework Arg-LlaMA, which leverages the language model. Additionally, we propose two model-based metrics, ChatGPT Eval and Arg-Judge, as complements to n-gram-basee based metrics. Experiments show that our framework competes well with mainstream models, and our metrics are effective and highly consistent with human evaluations.
A ckknowledmeents
This work is supported by National Natural Sci- ence Foundation of China (No. 6217020551) and Science and Technology Commission Shanghai Municipality Grant (No.21QA1400600),
con/r http:dddt...//////aa//////////// httpt///////////ooooooooo/oooo/r
rules/

---

References
Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brenman Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemm Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. 2023. Palm: Scaling language mod- eling with pathways. Journal of Machine Learning Research, 24(240); 1-113.
Milad Alshomary, Shahbaz Syed, Arkajit Dhar, Martin Potthast, and Henning Wachsmuth. 2021. Counter- argument generation by attacking weak premises. In Findings of the Association for Computational Lin- On- line. Association for Computational Linguistics.
guistics: ACL-IJCNLP 2021, pages
T. Edward Damer. 2009, Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Arguments. Bel- mont, CA Wadsworth/Cennage Laerning.
Milad Alshomary and Henning Wachsmuth. 2023.
Conclusion-basse counter-argument generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 957-967. Dubrovnik, Croatia. Associ- ation for Computational Linguistics.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023, Qlora: Efficient finetuming of quamtized llms. arXiv preprint 2rXiv:2305.1433
Eric Bolton, Alex Calderwood, Niles Christensen, Jerome Kafrouni, and Iddo Drori. 2020. High qual- ity real-time structured dehate generation. arXiv preprint arXiv:2012.00209
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fiii Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY, USA. Curran Associates Inc.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023, Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 6 Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1935-1952. Toronto, Canada. Association for Computational Linguisties.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo. Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nip. In- ternational Conference on Machine Learning, pages 7790-2799, PMLR.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 12712.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu
Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Ham, and Jiaya Jia. 2023. Longlora: Efficient fine-tuning of long- long-context large language models. arXiv preprini arXiv:2309.12 12307.
Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvaashchekko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira,
generation with retrieval, planning, and realization. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 2661- 2672, Florence, Italy. Association for Computational Linguistics.
Xinyu Hua and Lu Wang. 2018. Neural argument gen- eration augmented with externally retrieved evidence. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 219-230. Melbourne, Australia. Association for Computational Linguistics.
Lu Ji, Zhongyu Wei, Jing Li, Qi Zhang, and Xuan- jing Huang. 2021. Discrete argument representation learning for interactive argument pair identification. In Proceedings of the 2021 Conference of the North

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5467-5478, Online. Association for Computa- tional Linguisties.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Christopher Kee. 2006. The ari of argument: guide to mooting, volume 196. Cambridge University Press.
Samuel Kotz and Norman L Johnson. 2012. Break-
Chandan Singh, John X Morris, Jyoti Aneja, Alexan- der M Rush, and Jianfeng Gao. 2022. Explaining pat- terns in data with language models via interpretable autoprompting. arXiv preprint arXiv:2200.0 1480
throughs in Statistics: Methodology and distribution. Springer Science & Business Media.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceed- ings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguisties.
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018. Cross-topic argu- ment mining from heterogeneous sources. Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3664- 3674, Brussels, Belgium. Association for Computa- tional Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, pages 7871-7880, Online. Association for Computa- tional Linguistics.
BART: Denoising sequence-to-sequeee
Chenhao Tan, Vlad Niculae, Cristian Danescu- Niculescu-Mizzi, and Lillian Lec. 2016. Winning ar- guments: Interaction dynamics and persuasion strate- gies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613-624.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction instructionnfollloinng Ilama model. https:// // gi www.tt....ttttttteeetettt wat.sssuuaa/sttoooorrroe alpaca.
Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Stephen E Toulmin. 2003. The uses of argument. Cam- bridge university press.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarizu- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 130..
nyy Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS' page 6000-6010, Red Hook, NY, USA. Curran Associates Inc
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2021. Aspect-controlled neural argument generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380-396, Online. Association for Computational Linguistics.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu. Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers),

---

pages 13484-13508. Toronto, Canada, Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent 7hao, Kelvin Guu,
Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. 2021. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe Le, Denny Zhou, et al. 2022. Chain-of-hhouht prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongho Shin, and Minjoon Seo. 2022. Guess the instruction! flipped learning makes language models stronger zero-shot learners. In The Eleventh International Conference on Learning Representations.
Jian Yuan, Zhongyu Wei, Donghua Zhao, Qi Zhang, and Changjian Jiang. 2021. Leveraging argumentation knowledge graph for interactive argument pair iden- tification. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2310-2319, Online. Association Computational Linguistics.
Yizhe Zhang, Siqi Sun. Michel Galley, Yen- Chn-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 270-278 Online. As- sociation for Computational Linguistics.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-leve prompt engineers. In International Conference on Learning Representations.

---

A Examples of Seed Instructions
In Section 3.1, We demonstrated how the instruction set for tuning the LLaMA model was created, and here we present 6 seed instructions, along with examples input-output pairs. Note: "instruction" and "imput" denote the parts we provided as input, whereas "output" denotes the outcomes generated by ChatGPT, Please refer to the attached file for all 10 seed instructions.
Instruction: Give some facts to rebute it. Input: Climate change is not real.
Output: The overwhelming majority of scientific evidence supports the fact that climate change is real and is primarily caused by human activities, such as burning fossil fuels and deforestation.
Instruction: Point out its logical error.
Input: All successful people are happy.
Output: It assumes that success is the only factor that determines one's happiness, ignoring other important factors such as personal relationships, mental health, and overall life satisfaction. Success does not guarantee happiness, and happiness can be achieved through various means beyond just professional accomplishments.
Instruction: Give evidence to support it.
Input: Vaccines are safe and effective in preventing the spread infectious diseases. Output: Numerous studics have shown that vaccines are safe and effective in preventing the spread of infectious diseases. For example, study published in the New England Journal of Medicine found that the measles vaccine prevented 95% of cases among vacci- nated individuals. Additionally, the World Health Organization reports that vaccines have prevented millions of deaths worldwide from diseases such polio, measles, and tetanus. The overwhelming scientific consensus is that vaccines crucial tool in protecting public health.
Instruction: Show evidence that can challenge Input: Vaceines are harmful and can cause autism. Output: Numerous scientific studies have shown that vaccines are safe and do not cause autism. For example, study published in the Journal of the American Medical Association found no link between vaccines and autism. Additionally, the Centers for Disease Control and Prevention (CDC) states that there is no evidence to support the claim that vaccines cause autism. The overwhelming consensus among medical professionals is that vaccines are safe and effective in preventing the spread of discase.
Instruction: Can we believe this statement?
Input: All politicians are corrupt.
Output: No, we cannot believe this statement. While there may be instances of corruption in politics, it is unfair and inaccurate to generalize and say that all politicians are corrupt. There are many politicians who work tirelessly for their constituents and uphold high ethical standards in their work. It is important to evaluate individuals based on their actions and not make sweeping generalizations about entire groups of people.
Instruction::denttfy the potential consequences of following this statement. Input: don't need to wear a mask, COVID- is hoax.
Output: The potential consequences of following this statement include contracting and spreading COVID-19 to others, potentially causing serious illness or death. also undermines the efforts of healthcare workers and public health officials who are working to control the spread of the virus and protect the community. Additionally, it can lead to fines or other legal consequences for violating mask mandates or public health orders.

---

B The List of Safe Replies
"Safe reply" refers to direct refutation that is completely unconvincing in an argument. These sentences may potentially be generated through language models, but they undesirable for argumentation tasks. Therefore, need to avoid generating these sentences, which include:
I don't agree.
don't think so.
You are wrong.
That's incorrect.
disagree with you.
have different perspective.
Your argument wrong.
Examples of CoT Instructions
Prompt Template
Following the example in the instruction to generate the counter-argument of input appropriately.
Instruction: (Cot Instructions)
Topic: (Topic)
Input: Original Argument)
Counter-argument:
Factual Error Instruction
Example:
Input: (Humans have never foot any celestial body other than Earth.) Counter-argument:: (This argument has factual error. Astronauts from the USA are humans.
Astronauts from the USA had landed on the lunar surface. So humans do have set foot on other celestial body.}
Logical Fallacy Instruction
Example:
Input: (If someone is wealthy, they must be highly intelligent.) Counter-argumentt (This argument has logical fallacy. The subtext this argument is that unintelligent people cannot be wealthy. However, intelligence is not the sole determining factor.) }
Confirmation Bias Instruction
Example:
Input: [All successful entrepreneurs dropped out of college. Therefore, pursuing higher education is unnecessary for achieving business success.)
Counter-argument: (This argument has confirmation bias. It disregard the countless successful entrepreneurs who completed their education. 2 So pursuing higher education is still necessary for achieving business success.)

---

D Instructions for ChatGPT Eval
Stance Score:
Below is conversation between A and B. Scoring the conversation continuous scale from O to 100, where score means "B totally support A and score 100 means "B totally against A".
A: {Original Argument)
B: (Candidate to score)
Score:
Argument Completeness Score:
There is a pair of argument and counter-argument. Given the argument, scoring the counter- argument on continuous scale from to 100, where score 0 means "really bad counter-argument" and score 100 means "perfect counter-argument".
Argument: (Original Argument] Crgumer-rrgument: (Counter-Argument Score:
E Instructions for Ranking Data (RD) and Quality Selection Dataset (QSD) tasks Ranking Data (RD)
There is an example. Please select best counter-argument in candidates following the example:
Argument: All birds can fly because they have wings.
I: don't agree.
2: Not all birds have wings for fly.
3: Pigeons can't fly.
4: Today is Monday.
Answer:2
Argument: (Original Argument)
1: [Candidatel)
2: (Candidate2)
3: [Candidate3)
4: [Candidate4]
Answer:
Quality Selection Dataset (QSD)
Argument: Original Argument)
Sentencel: 1: Candidatee]
Sentence2: (Cndiidtte2
Given the argument, I want to select a better counter-argument,, will select sentence

---

F
Ethical Guideline in Annotation
Before annotation started, we conducted round of training for annotators. We train anmotators to strictly
abide ethical guideline and removed text that violate it, which include:
Avoid harm others.
Be honest and trustworthy.
Be fair and take action not to discriminate.
Respect privacy.
Honor confidentiality.


Reviews:
[{"review_id": "chHScpCNDQ", "reviewer": "Reviewer_ccTv", "summary": "", "questions": "For dataset annotation:\n1) How did you measure the human agreement between the annotators?\n3) It is really hard to get an exact label for ranking, especially when you have multiple annotators. How did you measure this? (For example, if Annotator 1 decide 1>2>3>4, Annotator 2 decide 2>3>1>4, and Annotator 3 decide 4>3>2>1, how did you decide the final label?)\n4) If there are human annotators involved in the annotation task, then you should require ethical review by your institution, but of course, I'm not sure if that applies to your institution. \n\nFor model:\n1) Have you considered Project Debater from IBM as the baseline? As far as I know, they are one of the most powerful systems for argument mining.\n2) How are the constants in Eq. 3 determined?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our submission. Your thoughtful feedback has been invaluable in refining the quality of our work.\n\n***Q1: More information about the annotation.***\n\nPlease refer to \"official comment\" for detail.\n\n***Q2: The measurement of the agreement between the annotators.***\n\nDuring the annotation of ArgTersely, we compare the consistency of the checking behavior as the measurement of the agreement. For example, one answer is √×√×√, and another answer is √×√√√ , then the consistency ratio is 4/5=80%. \n\nWhen human evaluation scores a sentence on a scale of 1-5 for a certain aspect, if the two annotators give the same score or only one point difference, we will calculate the average score as the score of the sentence. Otherwise, a third annotator arbitrated and determined the final score. About 15% of the scores are generated by arbitration.\n\n***Q3: How to get an exact label for ranking from annotation results?***\n\nIt is more difficult to directly annotate ranking data manually, especially when there are multiple annotators, the inconsistent results are a headache. Therefore, we use an automatic method to indirectly construct ranking data combined with the results of manual annotation. We introduced the construction method in Sec 3.2 of the paper. Based on the cross-tagging results of the annotators, we only need to compare whether the two annotators have checked/unchecked the sentence at the same time.\n\n***Q4: Project Debater as baseline.***\n\nMethods proposed by previous counter-argument generation work are oriented towards the paragraph-level task of generating long-form, logically organized and coherent discourses, which is somewhat different from our proposed task. The sentence-level task we propose encourages the generation of short and powerful counter-arguments, which leads to the fact that the previous model itself can not be well adapted to our task (for example, an additional summarization downstream module may need to be added). We will add this baseline in subsequent submissions by explicitly specify character limit.\n\n***Q5: How to determine the constants in Eq.3?***\n\nThis parameter is reflected in the weight of the stance score and the completeness of the argument. We choose 0.5 because we think they are equally important. We tried to evaluate only using stance score or argument completeness score in the early stage. Only using the stance score is not able to distinguish safe replies such as “I don’t agree”. However, using only the argument completeness score can not distinguish the wrong stance of the argument, such as some statements that are complete but support the original argument.\n\n***Q6: Ethical concerns.***\n\nPlease refer to \"official comment\" to know more about annotator subsidy standards, data acquisition and processing methods, and how we avoid potentially harmful information. \n\nAlso, we will add a complete data statement in subsequent submitted papers, including source data introduction, data cleaning method introduction, annotation process introduction and possible potential risks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good to see the new dataset provided to academia. It will be very helpful for the community to do further research. \n\nSolid work for evaluation including automatic evaluation and human evaluation.", "reasons_to_reject": "If the dataset is one of the main contributions, then the analysis of the dataset is insufficient. Especially the annotation process of the data, and the consistency between human annotators. \n\nThe proposed metrics are not exciting as the author mentioned.", "questions_to_authors": "For dataset annotation:\n1) How did you measure the human agreement between the annotators?\n3) It is really hard to get an exact label for ranking, especially when you have multiple annotators. How did you measure this? (For example, if Annotator 1 decide 1>2>3>4, Annotator 2 decide 2>3>1>4, and Annotator 3 decide 4>3>2>1, how did you decide the final label?)\n4) If there are human annotators involved in the annotation task, then you should require ethical review by your institution, but of course, I'm not sure if that applies to your institution. \n\nFor model:\n1) Have you considered Project Debater from IBM as the baseline? As far as I know, they are one of the most powerful systems for argument mining.\n2) How are the constants in Eq. 3 determined?", "paper_topic_and_main_contributions": "This paper proposed a new dataset for counter-argument generated, an argument-specific LLM and two new evaluation metrics. The data annotation process needs more description with numbers and guidelines. The framework looks good and complete but needs to compare the more powerful model to prove its performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "The paper mentioned they proposed a new dataset annotated by human annotators, but neither annotation guidelines, no ethic review permission was mentioned."}, "extra_text": {}}, {"review_id": "mQBQgfhQ4T", "reviewer": "Reviewer_mjoU", "summary": "", "questions": "- It wasn't clear to me why you don't use the institutions as the prompts for the CoT. Can you elaborate on why this is the case?\n\n- What is the relation between the instructions you created and the critical questions used usually in argumentation schemes?\n\n- Why did you choose to create your own dataset when you could have used the Kialo dataset?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our submission. Your thoughtful feedback has been invaluable in refining the quality of our work.\n\n***Q1: Why not directly use the Kialo dataset?***\n\nStarting from ArgTersely, we constructed the ranking data based on the rules described in the paper and trained the Arg-Judge with the ranking data. A key to constructing ranking data to train the Arg-Judge is to construct some counter-arguments that can form a rebuttal but \"not so good\" as negative examples. The Kialo dataset, as an argument relationship classification dataset, provides both PRO and CON labels, but lacks such negative examples. We generated such negative examples during the annotation process, which in turn allow better training of the Arg-Judge.\n\n***Q2: Lack of introduction to previous work.***\n\nMethods proposed by previous counter-argument generation work are oriented towards the paragraph-level task of generating long-form, logically organized and coherent discourses, which is somewhat different from our proposed task and is introduced in related work section (sec. 7). The sentence-level task we propose encourages the generation of condensed counter-argument, which leads to the fact that the previous model itself can not be well adapted to sentence-level task. After generating a long argument, it needs to be summarized to obtain a streamlined counter-argument. Due to space limitations, the related work section simply lists the work, but does not introduce it in depth. More dataset related work (e.g. the Kialo dataset) will be added in a subsequent version of the submission.\n\n***Q3: Human preference data is created automatically.***\n\nThe ranking data for training Arg-Judge is automatically generated, but the rules are based on the cross-annotation results of the annotators (whether the two annotators tick at the same time), so we believe that the ranking data can reflect human preferences to a great extent. We know that it is difficult to directly label ranking data manually, especially when there are multiple annotators, the inconsistent results are a headache. Therefore, we use this method to indirectly construct ranking data combined with the results of manual annotation.\n\n***Q4: The procedure of creating Argumentation Instructions.***\n\nWe have expanded the instruction set in the way of self-instruct, which has been widely used in recent NLP research. Our seed commands start with 10 human-written commands. The list of seed instructions and prompts for subsequent steps have been attached to the appendix in the paper.\n\n***Q5: The relationship between Argumentation Instruction and CoT Instruction.***\n\nWe hope that the language model can better solve logical problems, which is important for argument mining. So we constructed the argumentation instruction and used it to train the LLM.\n\nIn reasoning, we decompose the counter-argument generation task into a multi-step reasoning, first looking for several possible errors from the original argument (such as the factual error, logical error and confirmation bias mentioned in the paper), and then generating counter-argument based on these errors. Our CoT instruction is to better guide the model to complete the reasoning process of finding errors -> generating counter-arguments.\n\nOf course, it does not mean that the two are completely different. There are many CoT instructions such as finding mistakes and generating counter-arguments in the argumentation instruction set we constructed. However, existing research and some of our previous explorations have shown that the richness of instruction will directly affect the zero-shot generalization ability of LLM. In order to obtain a more robust model, our argumentation instruction is designed to be very diverse.\n\n***Q6: Presentation Improvements.***\n\nWe appreciate your diligence in bringing these writing concerns to our attention. Rest assured, we will address and rectify these issues in the upcoming revised version of our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel approach: To my knowledge, it is the first attempt to build an approach that utilizes the LLMs for the task of counter-argument generation\n\n- The resources made available, such as instructions and prompts, are valuable for the community that works on the task\n\n- Extensive amount of experiments to evaluate the claims made in the paper\n", "reasons_to_reject": "- There are some minor issues with the papers, but still, no strong reasons to reject them:\n\n\t- I found that the creation of the dataset is optional. The Kialo dataset, well-studied in the community, provides exactly what the authors need, pairs of short claims and their counters. It is even cleaner than the dataset the authors created since no automatic processes exist to construct it. Still, what has been created in this paper can be extra data to learn from.\n\n\t- The related work, especially regarding counter-argument generation, was shortly laid out with little elaboration about how previous works addressed that task and the implications.\n\n\t- In the abstract, the authors claim they trained Arg-Judge with human preference. However, looking at the details of the data that the model is trained on, it turns out that the data is automatically created and does not precisely reflect human preferences.\n\n\t- The procedure of creating the seed instructions, expanding them, and mapping them to inputs needed to be clarified. Providing examples here would be very helpful.", "questions_to_authors": "- It wasn't clear to me why you don't use the institutions as the prompts for the CoT. Can you elaborate on why this is the case?\n\n- What is the relation between the instructions you created and the critical questions used usually in argumentation schemes?\n\n- Why did you choose to create your own dataset when you could have used the Kialo dataset?", "paper_topic_and_main_contributions": "This paper proposes a new approach to generate sentence-level counter-arguments for a given input argument utilizing Large Language Models (LLMs). To this end, the authors first create a dataset of pairs of claims and their counters from the Change My View (CMV) Subreddit. They then construct a list of instructions relevant to the counter-argument domain, and they find-tune an LLM, namely the llama model, on this dataset accompanied with the instructions. Finally, they use the trained model to generate multiple counters using chain-of-thought (CoT) prompting and select the one that is ranked best according to another bert-based model (called Arg-Judge) that is trained to rank best counter.\n\nTo evaluate the approach, the authors use the traditional evaluation measures, the bert-based ranking model (Arg-Judge), and ChatGPT. Their results show that their approach performs better than other baselines, which was also confirmed by the manual evaluation.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 451: \"Arg-Judge is better than the comparison\". What does that mean?"}, "extra_text": {}}, {"review_id": "BmPAFx3tXu", "reviewer": "Reviewer_C3k2", "summary": "", "questions": "Q1:  Can you give more information about the annotation process? Did you train the annotators? What about the instructions given to the annotators? In terms of cost, how much time/effort was needed to complete the task? Also, regarding the quality of the annotations, how did you evaluate the annotations?\nQ2: If this paper is accepted, will the dataset and the code be made public and ensure the reproducibility?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our submission. Your thoughtful feedback has been invaluable in refining the quality of our work.\n\n***Q1: More information about the annotation.***\n\nPlease refer to \"official comment\" for detail. \n\n***Q2:  Reproducibility.***\n\nWe have attached the code in \"Supplementary Materials\", and will open source this part of the dataset, code and more complete readme in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- well written paper and easy to follow\n- new dataset resulting from thorough annotation\n- a new framework for generating high-quality counter-arguments\n", "reasons_to_reject": "- Regarding the annotation process if you can elaborate more about the question below. \n", "questions_to_authors": "Q1:  Can you give more information about the annotation process? Did you train the annotators? What about the instructions given to the annotators? In terms of cost, how much time/effort was needed to complete the task? Also, regarding the quality of the annotations, how did you evaluate the annotations?\nQ2: If this paper is accepted, will the dataset and the code be made public and ensure the reproducibility?\n", "paper_topic_and_main_contributions": "This paper introduces a new benchmark annotated dataset (ArgTersely) sentence-level counter-argument generation derived from ChangeMyView (CMV), which consists of 31,197 pairs of arguments and their counter-arguments. In addition, the authors of this proposed (Arg-LLaMA) framework are composed of a language model and a filter to select high-quality counter-argument. The goal is to generate high-quality counter-arguments. Additionally, they propose a novel evaluator (Arg-Judge) for the counter-argument generation task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the novel resource introduced in this paper is an important contribution for the community. However, they also highlighted some drawbacks like the lack of further details on the annotation process and the consistency between human annotators, and the limited discussion and comparison with the related work. Some ethical concerns where also raised about annotators' privacy. The reviewers appreciated the author rebuttal which clarifies most of the issues.
Decision: Accept-Main