ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization
Authors: Xiutian Zhao
Ke Wang
Wei Peng
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue dataset
long dialogue summarization
target-independent stance detection

Abstract: Dialogue agents have been receiving increasing attention for years, and this trend has been further boosted by the recent progress of large language models (LLMs). Stance detection and dialogue summarization are two core tasks of dialogue agents in application scenarios that involve argumentative dialogues. However, research on these tasks is limited by the insufficiency of public datasets, especially for non-English languages. To address this language resource gap in Chinese, we present ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization. Our dataset consists of 1,218 real-world debates that were conducted in Chinese on 476 unique topics, containing 2,436 stance-specific summaries and 14,133 fully annotated utterances. Besides providing a versatile testbed for future research, we also conduct an empirical study on the dataset and propose an integrated task. The results show the challenging nature of the dataset and suggest a potential of incorporating stance detection in summarization for argumentative dialogue.

ORCHID:
A Chinese Debate Corpus for Targee-Independet Stance Detection and Argumentative Dialogue Summarization
Xiutian Zhao' Ke Wang Wei Peng"
Huawei IT Innovation and Research Center
(zhaoxiutian, wangke215, peng. wei wwwwweeee...ommm com
Abstract
Debate whether technology is ethically neutral?
Pro
Dialogue agents have been receiving increas- ing attention for years, and this trend has been further boosted by the recent progress of large language models (LLMs). Stance detection and dialogue summarization are two core tasks of dialogue agents in application scenarios that involve argumentative dialogues. However, research on these tasks is limited by the in- sufficiency of public datasets, especially for non- non-Eng mon-English languages. To address this lan- guage resource gap in Chinese, present OR- CHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization. Our dataset consists of 1,218 real-world de- bates that were conducted in Chinese on 476 unique topics, containing 2,436 stance-specific summaries and 14,133 fully annotated utter- ances. Besides providing versatile testbed for future research, we also conduct an empirical study on the dataset and propose an integrated task. The results show the challenging nature of the dataset and suggest potential of incor- porating stance detection in summarization argumentative dialogue.'
It depends whether tech has biases that lead differential outcomes...
Con
...tech is not conscious, it cannot TAKE neutral position...
Tech has no position...
...yet NO position is not NEUTRAL position...
...a few rounds later
Pro Sum.
is the USER's good and evil that gives tech ethical values, not tech itself...
Con Sum.
A gun ethically different from a wheel, for the former tech tends to destroy...
Figure I: excerpt of one debate ORCHD,, One
debate entry of our dataset consists of: (1) debate topic, (2) position statements of both sides, (3) utterances la- belled with speaker and stance, and (4) stance-specific summaries. Original text is in Chinese (see Appendix G for more complete example).
1 Introduction
Recent development large language models (LLMs) have pushed the general interest on di- alogue agents to new level, and increasingly powerful LLMs such as GPT series demonstrated promising capabilities across multiple application scenarios. Among various tasks that have been assigned to dialogue agents, engaging argumenta- tive dialogues (Macagno, 2000; Walton, 2008) has long been challenging one. Regardless of spe- ciñic aims, whether winning a debate (Zhang et al., 2016), convincing people (Prakken et al., 2020), or
opening up minds (De Kock and Vlachos, 2021: Farag et al., 2022), dialogue agents rely on two foundation abilities: stance detection and summa- rization. Stance detection aims to reveal attitudes of arguments, and the goal of summarization is to collect and condense information in order to build arguments. They collaboratively support compre- hending and developing arguments, consequently both abilities are crucial for engaging argumenta- tive dialogues (Chen et al., 2017: Lawrence and Reed, 2019; Wang and Wan, 2021; Sanders et al.,
'Equal contribution. "Corresponding author
'ORCHiD is publicly available at petpp:///githul com/ xiutian/OrChii

---

2022).
research, we present ORCHID (Oral Chinese Debate), to the best of our knowledge, the first Chinese debate summarization dataset annotated with multi-granularity and stance-specific sum- maries, also the first Chinese benchmark for target- independent stance detection. Our dataset con- sists of 14.133 fully annotated utterances and 2,436 stance-specific summaries from 1,218 real-world debates that were conducted in Mandarin. We employed an automatic speech recognition (ASR) to transcribe raw data, followed by manual post- correction and annotation. We provide debate sum- maries on two-levels of granularity, short concise statements and long comprehensive summaries of both stances. Stances and debaters are labelled at utterance level. Furthermore, we conduct a pre- liminary empirical study the constructed data. Sharing this novel dataset, we hope to resupply the research community tackling tasks including dia- logue summarization, stance detection and other argument mining tasks.
In general natural language processing (NLP), stance detection is classify the stance ("favor', "against' and 'none") of piece of comment with respect to a target entity or claim (Hasan and Ng, 2013; Nguyen et al., 2016; Küçükk and Can, 2020; Hardalov et al., 2022). The other task, text summa- rization aims to compress information from large piece of text and produce concise and comprehen- sible digest (Gillick al., 2009; Shang et al., 2018), Dialogue summarization, fittingly, takes dialogues as source text.
However, some unique features of argumenta- tive dialogues propose atypical challenges for the two tasks. Regarding summarization tasks, argu- mentative dialogues, such as debates, meetings and online forum discussions, often contain contradic- tory utterances with conflicting stances (Zou et al., 2021), making them more convoluted to summa- rize. Also, in comparison with written text, spo- ken dialogues naturally carry more noises such as mispronounces, rephrasing and repeated words that obstruct summarizaiton. Mcanwhile, unlike typical target-specific stance detection whose tar- gets are explicit entities (c.g.. 'Metaverse'), stance detection argumentative dialogues is target- independent, meaning that the targets of those studies are claims in the form of complete sen- tences (e.g., 'The commercial value of metaverse is overestimated")
In summary, our contributions are three-fold: (1) introduce ORCHID, the first Chinese dataset for debate summarization and target-independent stance detection; (2) we propose new integrated task, seance-speciffe summarization, which is sug- gested by the experiment results to improve sum- marization on argumentative dialogues; and (3) we conduct preliminary experiments, benchmarking classical and newly-suggested tasks against our dataset, reporting corresponding results, and set- ting initial baselines for future work.
Despite the progress made on the tasks, research community's s effort has been decelerated by an in- sufficiency of proper language resources. Existing dialogue summarization datasets are dominantly in English (Gliwa et al., 2019; Durmus and Cardie, 2019; Roush and Balaji. 2020; Fabbri et al., 2021; Chen al., 2021). Regarding non- -on-English sum- marization datasets, prior resources in Chinese are either focus on one specific domain (Song al., 2020; Zou et al., 2021: Lin et al., 2021; Huang et al., 2020), or without stance-specific summaries (Feng et al., 2022). There still is lack of multi-domain and annotated Chinese dialogue summarization datasets. Moreover, most existing stance detection datasets are target-specific regardless of language (Alturayeif et al., 2023). Overall, in terms of Chi- nese language resourees, the amount of datasets suitable for argumentative dialogue summarization is highly limited, and there currently no bench- mark for target-independent stance detection.
2 Related Work: Existing Datasets
We reviewed existing dialogue summarization and stance detection datasets as presented in Table 1 and 2. For dialogue summarization datasets, we observe major amount imbalance between argu- mentative ones and con-argumentative ones. Also, argumentative dialogue summarization corpora are primarily meeting transcripts (Kumar and Kabiri, 2022), and non-English ones are rare. Stance de- tection studies suffer from a lack of stance de- tection datasets in Chinese, in particularly target- independent ones.
2.1 Stance Detection Datasets
Most previous stance detection studies can be grouped into two categories by target-comment dependency: (1) sprect- and (2) target- independent (Küçük and Can, 2020). While much
To remedy this shortage and facilitate related

---

Content
Content
Argumen
Dialogue Topic
Summary
Dataset
Lg.
Dialogue
Domain
Type
-tative
Types
SAMSum
En
16,369
multiple
fictitious chat logs fictitious chat logs customer service logs
X
abs.
MSAMSum
4"
5.930
multiple
X
X
abs.
CSDS
Zh
10,701
single
X
abs. role-oriented
SportsSum
Zh
5,428
single
sports commentaries
X
abs. (news article) abs.
DualogSum
En
13,460
multiple
daily conversations
X
X
AMI
En
137
single
meetings
abs. ext. abs. abs. (query-paired)
ConvoSumm
En
500
multiple
online comments
QMSum
En
232
multiple
meetings
ELITR
2
179
single
meetings
abs. (minute)
VCSum
Zh
239
multiple
meetings
abs. & segmented
ORCHID
Zh 1,218
multiple
competitive debates
abs. & stance-based
Table 1: Comparison of some existing dialogue summarization datasets. Lg. denotes language: for English and 7h for Chinese. Abs. and Ext. denotes abstractive and extractive summary. Dialogue Topic indicates whether each dialogue has headline or title. "MSAMSum contains paraliel corpora Chinese, English, French and Russian. Part of DialogSum dialogues are argumentative. 'ELITR includes 59 Czech and 120 English meetings.
Target
Content Type
Spoken I Language
Dataset
Lg.
Comment
Stance Labels
Type
Dep.
SemEval-200
En 4,163
5
Entity
TS
Favor, Against, None
social media posts social media posts social media posts social media posts website comments
X
NLPCC-2016
Zh
4,000'
Entity
TS
Favor, Against, None
CSD
Zh 5,876
Entity
TS
Favor, Against, Neither
X
MTSD
En
4,455
4
Fintity
MT
Favor, Against, None
X
X-stance
3b
67,271
Entity
CT
Favor, Against
VAST
En
23,525
5,634
Entity
CT
Pro, Con, Neutral
news article comments
X
Emergent
En
2,595
300
Claim
TI
Favor, Against, Observe
articles Wikipedia articles
X
IBM Debater
2,394
55
Claim
T
Pro, Con
Perspectrum
En
11,164
907
Claim
TI
5 lahels
website comments
X
IAM
En 4,890
123
Claim
TI
+1,-1
Wikipedia articles
X
ORCHID
Zh 16,529
2,436
Claim
TI
Pro, Con, Mixed
competitive debates
Table 2: Comparison of some existing stance detection datasets. Type denotes whether the targets entities or claims. Dep. denotes the dependency of target (7S: target-specific, MT: multi-target, CT: cross-target, 77: target-independent). "NLPCC-2016 consists additional 2,400 unannotated comments. *X-stance includes cases in French, German and Italian. "Perspectrum has support, mildly-support, perspective for labels. 11,113 debating utterances plus 2,436 closing remarks.
support, oppose, mildly-oppose and not valid
rarer, datasets of two other dependency types have also been proposed: (3) multi-target (e.g., Sobhani al. 2017; Barrière et al. 2022) and (4) cross- target (e.g., Vamvas and Sennrich 2020; Allaway and McKeown 2020) as Alturayeif et al.'s (2023) survey curated. We summarised a collection of them in Table 2 for comparison.
task similar to SemEval- 6016 and released dataset 4,000 Chinese microblogs. CSD (Li al., 2022) is newly released and contains 5,876 labelled website comments in Chinese on COVID-19 vaceination.
Ingeet-Indeeedeent Datasets Turning now to target-independent stance detection datasets, Fer- reira and Vlachos (2016) introduced the 'Emer- gent' dataset that consists 2,595 comments (news article headlines) 300 claims. IBM Debater (Bar-Haim al., 2017) leverages 2,394 Wikipedia articles and labelled their stances on 55 claims. More recently, Chen al. (2019) collected 11,164 website comments on 907 claims. IAM (Cheng et al., 2022) is also created by sourcing Wikipedia
Prior studies are primarily on target-specific stance detection, and the sources are centered on so- cial media posts. SemEval-2016 (Mohammad et al., 2016) introduced stance detection shared task on a dataset of 4,163 tweets. A few more shared task datasets were created (Derczynski al., 2017; Gorrell et al., 2019). Regarding Chinese datasets, NLPCC-2016 (Xu al., 2016) presented a shared

---

articles. In addition, Durmus and Cardie (2019) constructed large English online debating corpus with stance labels. Besides language, our dataset differs from it for having additional 'mixed' stance, stance-specific summaries, and spoken style text (Durmus and Cardic's (2019) online debates are not oral ones but by writing threads).
corpus that consists of 120 English and 59 Czech meetings.
Despite the work made so far, there remains great imbalance between English dialogue sum- marization datasets and non-English ones. We also sense urgency for expanding the diversity of argumentative dialogue corpora beyond meeting transcripts.
To the extend of our knowledge, there currently is no Chinese target-independent stance detection dataset available. Also, the source texts of existing stance detection datasets are dominantly of writ- ten rather than spoken, so our spoken-style corpus could be rare supplement to the language resource pool.
3 Creating ORCHID
Having reviewed existing stance detection and di- alogue summarization corpora, we determine to construct new versatile dataset that adapts to both tasks. To this end, we introduce ORCHID (Oral Chinese Debate), as the name implics, a corpus that features oral debates in Chinese.
2.2 Dialogue Summarization Datasets
Dialogues range from daily chitchat to formal de- bate, and researchers have introduced diverse dia- logue summarization datasets. SAMSum dataset (Gliwa al., 2019) greatly accelerated this field by proposing large-scale dataset of fictitious chat- dialogues created by linguists. English data re- sources that leverage other forms of dialogues have also emerged: online forum posts (Khalman et al., 2021), interviews (Zhu et al., 2021),, debate evi- dence documents (Roush and Balaji, 2020), daily conversations (Chen et al., 2021), and screenplay transcripts (Chen et al., 2022).
We select oral debates in competition scenario as our source for following reasons: (1) debates are highly argumentative and thematic, and stances of both sides are clearly stated and not subject change: (2) debate utterances are high-quality in terms of logic and rhetoric (Zhang et al., 2016). and such utterances are much less colloquial than daily conversations while retaining partial oral fea- tures; (3) since existing stance detection corpora are predominantly written texts, utterances of spo- ken styles and expressions, which debates offer, could be valuable addition.
Although Chinese datasets in dialogue summa- rization are scarce, some valuable corpus were in- troduced, including multilingual fictitious chat dataset derived from SAMSum dataset (Feng et al., 2022); medical conversation (Song et al., 2020); sports live commentaries (Huang et al., 2020); cus- tomer service logs (Lin et al., 2021; Zou et al., 2021).
The construction of our dataset consists four ma- jor stages: data collection, ASR-aided transcrip- tion, manual annotation, and quality control. We first collect videos from public sources. Next, we employ an ASR system to obtain raw transcripts, followed by automatic labelling stances and man- ual correction. Finally, we extract and construct stance-specific position statements and conclusive summaries. We also apply several quality control measures during the process.
Argumentative Dialogue Summarization Datasets To reduce the scope, previous English corpora of argumentative dialogue summarization have focused heavily on meetings. AMI (Janin et al., 2003) and ICSI (Carletta et al., 2005) are two early and widely-used meeting corpora. More recently. ConvoSumm (Fabbri et al., 2021) presents 500 conversations of multiple types: news article comments, discussion forums, community question answering and email threads. QMSum (Zhong al., 2021) uniquely developed 1.808 query-summary pairs on 232 meetings. Wu al. (2023) introduced Chinese dataset on 123 meetings annotated with segment-wiss, wise summarics. ELITR (Nedoluzhko et al., 2022) is a minuting
3.1 Data Collection
An pilot screening on publicly accessible data reveals that there is an absence official tran- scripts for most debate competitions, so we alter- natively utilize original debate videos our pri- mary sources. From major video sharing platforms where the competition organizers regularly release match videos (see Appendix E), harvest a tal of 1,704 debate videos across 59 competitions that were conducted in Mandarin Chinese. After an elaborate filtering process (see $3.4), we end up retaining videos from 1,218 debates across 30

---

competitions.
program scripts obtain preliminary segmenta- tion and labelling, a manual correction is followed to correct wrongly split or labelled utterances.
Economy & Business
13
Health & Evironment
79
Education Profession
Topic and Position Statement We extract topics from the debate adjudicators' pregame introduc- tions that we retain in the transcription process. addition, our annotators classify debate topics into eight domains: education profession, science & technology, philosophy & ethics, politics & law, culture & society, art & entertainment, economy & business and health & environment as shown in Table 2. Since there are topics involving multiple domains, the number of labels is not restricted to one.
Art Entertainment
88
Science Technology
147
Culture & Society
179
98
Politics & Law
Philosophy Ethics
Figure 2: Distribution of topic domains in ORCHID, While there are 476 unique topics, one topic could be classified into multiple domains.
Conventionally, debate competitions in English world phrase their topics (or 'motions') into con- firmatory assertions (e.g.. "Technology is ethically neutral"), which naturally are position statements. However, most competitions in Chinese world phrase their topics binary choice questions (e.g., 'Whether technology is ethically neutral?'). There- fore, our annotators manually rephrase question- style topics into position statements for both sides.
3.2 ASR-aided Transcription
We employ a commercially established automatic speech recognition (ASR) system developed by iFLYTEK (see Privacy and Licensing) to obtain raw machine transcription, then our annotators manu- ally post-correct any lexical error or wrongly put punctuation. We generally follow Mirkin et al. 's (2018) pipeline to conduct the ASR-aided transcrip- tion from source videos texts.
Debate Segmentation Debate adjudicators' an- nouncements often contain specific signal phrases (e.g., 'Let now invite the Third Proposition Speaker to rebuttal') that can serve as cound-change indicators. We automated the preliminary seg- mentation by utilizing the announcements via pro- grammed scripts. Next, the annotators correct missed or wrongly split utterances.
To more specific, firstly, we apply the ASR system on audios to obtain raw machine transcript. Secondly, the contents other than debaters' utter- ances, such as accidental interruptions and post- debate comments, are discarded. However, we preserve debate adjudicators' announcements, such pregame introductions and topic statement, for further utilization in the annotation stage. Finally, our annotators manually proofread and correct any lexical and syntactic errors appeared, including wrongly recognized words and misplaced punctua- tion to create a fully-cleaned transcript.
Stance and Debater A typical debate match con- sists multiple monologue rounds and discussion (or 'rebuttal') rounds. An utterance in a debate match refers to a piece of cime-constrained speech that contains multiple sentences, uttered by either one side (monologue rounds labelled by 'pro' 'con') both sides (discussion rounds labelled by 'mixed').
3.3 Automated and Manual Annotation
Although we have considered separating the sen- tences of discussion rounds by pro and con sides, yet we turned down the separation for two rea- sons: firstly, automatic methods of speaker diariza- tion (Park et al., 2022) yielded unacceptable results (accuracy less than 60%), and human handpick approach at sentence level would be highly labor- intensive; more importantly, we argue that dis- cussion containing utterances from both sides a complete linguistic unit. Singling out and concate- nating sentences from one side likely to yield
The annotation consists of four major steps: (1) extracting topics and rephrasing them into position statements, (2) segmenting debates by utterances, (3) labelling utterances with stance and debater, and (4) post-editing transcripts to produce reference summaries.
The annotation mostly fact-grounded, since the truthfulness of topic extraction and utterance labelling can be verified by checking with original videos and hence unanimously agreed upon. We

---

incoherent and inconsistent contents. Therefore, we keep the integrity of discussion rounds and la- bel them 'mixed'.
inter-annotator agreement. We obtain Cohen's K close denoting very high agreement between the two collective annotators. In fact, there is only differences where the two groups initially dis- agreed upon, and both groups reached consent after round of review. The almost perfect agreement is expected since the labelling is fact-grounded.
Again, the annotators utilize the adjudicators' announcements to label segmented utterances with stance and debater. The announcements are re- moved after the annotation, retaining only the ut- terances of debaters.
Regarding the reference summary construction, to avoid personal style preference and bias, we instructed annotators to minimize their editing by limiting changes to removing greetings and shifting addressing to third-person (c.g., from 'we/l/you' to 'the pro/con side"). Additionally, annotators were randomly selected to double-check the correctness and overall consistency of post-editing made by other annotators.
Reference Summaries We construct reference summaries two levels of granularity: (I) short and concise position summaries; and (2) long com- prehensive stance-specific summaries.
The short stance-specific summaries are created by directly adapting position statements. for in- stance, if the pro side position statement was 'tech- nology is ethically neutral', then the short pro- specific summary would be "The pro side argues that (pro side position statement)' By concatenat- ing stance-specific summaries from both side, get short overall summary: "The pro side argues that (pro side position statement). and the con side argues that [con side position statement).'
3.5 Dataset Overview
The resulting dataset is summarized in Table 3. There are a pair of stance-specific summaries and an overall summary contains both stance-specific summaries for cach debate. Hence, 1,218 debates have a total of 2,436 stance-specific summaries and 1,218 overall summaries. It is to be noted that the statistics may be subject to change (see Appendix B).
The long stancc-specific summaries are derived from the closing remarks of debates. At the last round of formal debate, a debater from each side is expected to provide summative and compre- hensive remark that cover key statements and ar- guments of their team. we asked our annotators to manually post-edit (e.g., remove greetings and change first-person to third-person) them with min- imal changes to the contents (see $ 3.4). The long overall summaries are created following the similar pattern as their short counterparts: "The pro side argues that (pro-specific summary). ). The con side argues that (con-specific summary)."
Statistics
Value
# Total debates
1,218
# Total unique topics
476
# Total ulterances
14.133*
# Total stancc-specific summarics
2,436
Avg. debate length
18,349
Avg. utterances per debate
11.5*
Avg. utterances length
1,265*
Avg. stance-specific summary length
1,901
3.4 Quality Control
Table 3:
Overview statistics for ORCHID. Debate, ut- terance and summary average lengths are measured Chinese characters. "Closing remarks were excluded from calculation.
In order to ensure the quality of the dataset, filter out videos by following criteria: (1) drop videos that lack complete debate contents; (2) any video without human-reed reconnizaeee audio is also discarded because manual post-correction on such video was impracticable; (3) non-standard matches are also neglected (see Appendix A).
Benchmark and Results?
Having constructed ORCHID, we conduct an em- pirical study benchmark the performances of some existing methods on three challenging tasks against our dataset: (1) stance detection; (2) ab- stractive summarization; and (3) stance-specific
During the stance annotation is done indepen- dently by two groups of annotators of four (ran- domly selected from the pool of 12). Members within one group are required reach unanimous decisions on all instances, so the two groups can be viewed as two collective annotators. Hence, we calculate Cohen's 's K (Cohen, 1960) to evaluate
"The reported results are based on the 2023-06-15 snapshot of the dataset, hence the statistics in this section does not align with the ones in $ 3, for we further expanded and updated the datasct afterwards (see Appendix B).

---

summarization, new integrated task that we pro- pose.
the train set, adjusting syper-parametes using the validation set, run random seeds on the test 3 times, and report the average results.
To address above tasks, we split our data as summarized in Table 4. For Task 1, a total of 14,091 labelled utterances are split by roughly 78%/1 1 90/1 11%. To avoid over-fitting, we make sure that utterances from debates with the same topics do not appear in both train and test sets.
Considering that autoregressive LLMs have demonstrated unprecedented performance in many NLP tasks recently, we also devise two direct prompting methods based on GPT-3.5 (OpenAI, 2022): (3) Zero-shot Prompting, a direct prompt- ing method with minimal instructions and (4) Few- shot Prompting (Brown et al., 2020) that adds a few utterances (three in our case) with conrectly classified stances as examples in prompting (see full prompts can be found in Appendix F). Only the test set is used, and we run 3 times and report the average results.
Train
Dev.
Test
Total
Stance Detection
Pro utterances
3,321
510
514
4,345
# Con utterances
3,324
511
518
4,353
# Mixed utterances
4,360
513
518
5,391
# Total
11,005
1,534
1,550
14,091
Abstractive Summarization
As summarized in Table 5, direct prompt- ing methods on autoregressive LLMs (GPT-3.5) outperform fine-tuned bidirectional models, BERT and RoBERTa, by a large margin. This may par- tially due to the complexity of the task and the limited training data available. In addition, we observe that few-shot prompting boosts GPT-3.5 further on the task. Interestingly, the F1 scores on 'Mixed' label are better than other labels, which may suggest that it is easier to detect conflicting features in an utterance than identifying a particular stance.
Results
# Overall Summaries
828
104
104
1,036
Stance Stance-specific Summarization
# Pro summaries
828
104
104
1,036
# Con summaries
828
104
104
1,036
# Total
1,656
208
208
2,072
Table 4: Data split statisties for benchmark testing.
4.1 Task 1: Stance Detection
Task Definition Let D = {U, = (c,, | be a debate of n utterances. Each utterance consists of a piece of text ej (utterance) and a stance s;. Let t be designated target claim (a position statement). Given f and q, the task aims to predict stance & (pro, con, mixed) for each U.
Method
Acc.
Fi-Pro
Fi-Con
Fi-Mixed
Fine-tuning Methods
MacBERT-base RoBERTa-WAm--
56.35
50.37
49.21
50.47
72.13
68.36
67.18
70.84
Direct Prompting (GPT:3.5) Methods
Experiment Setup As shown in Table 4, stances labels for utterances are imbalanced 9/13%%//88% for 'Pro'/'Con'/ /'Mixedd" of total utterances). Hence, The task is a 3-way classifi- cation with imbalanced data, each utterance con- sisting one single stance label. Following Cheng al. (2022), we report both overall accuracy and per-class (stance) Fi scores.
Zero-shot Few-shot
82.21
78.50
78.86
79.68
88.71
85.77
84.72
88.13
Table 5: Results of target-independent stance detection on ORCHID dataset. Acc. denotes overall accuracy.
4.2 Task 2: Abstractive Summarization
We experiment on two well-established pre-
Task Definition Let D) = U..,,,,..... .,,,...,,.. be a debate of n utterances (excluding final remarks). The goal of abstractive summarization is to produce a summary Yoverall for given debate D.
trained models: (1) BERT (Devlin et al., 2019) and (2) RoBERTa (Liu et al., 2019). Specifically, we implement Mac base (Cui et al., 2020), an im- proved BERT with masked language model (MLM) correction pre-training task, which mitigates the discrepancy of pre-training and fine-tuning. Re- garding RoBERTa, we use RoBERTa-wwm-
Experiment Setup We follow previous works (See et al., 2017: Gliwa al., 2019; Roush and Bal- aji, 2020) to earry out our experiments. We choose the well-established ROUGE (Lin, 2004) scores as automatic evaluation metrics and report standard
(Cui
et al., 2021), a Chinese pre-trained BERT with whole word masking. We fine-tune the models on

---

Fi scores of ROUGE-I, ROUGE-2 and ROUGE- L. We take overall summaries as gold references ygold_ goldd_overal-- overall. Besides automatic evaluation, we also conduct a human evaluation on generated sum- maries.
utterance. Repeat the process until all utter- ances are viewed by the model (Zhang et al., 2023).
Results We report both automatic metrics and overall human evaluation scores in Table 6. The evaluators were requested to rate the generated sum- maries by four aspects: conciseness, fluency, faith- fulness and informativeness. An overall average is calculated.
We benchmark two classic extractive meth- ods, three fine-tuning abstractive methods and three direct prompting (specifically 50t-3.--turb-1116 66-06133 methods on our dataset. However, our single debate length (18,107 Chinese characters per debate in average) excesses the input size limits of most pre-trained models. We heuristically propose several approaches to address this issue (See full prompts in Appendix F):
We observe that the abstractive methods (HM- Net, Summ and DIALOGLM) fine-tuned on our dataset generally perform better than the extractive models Lead-3 and TextRank-3. Undoubtedly, the GPT-3.5 direct prompting approach exhibits the most satisfactory performances. Although direct LLM prompting methods yielded better results than fine-tuning methods, there is still large room for improvement. Among three prompting methods, Iterative Revision achieve the best overall results.
Lead-K: a widely-used simple baseline taking leading & sentences (See et al., 2017). We set to for benchmarking.
TextRank-K (Mihalcea and Tarau, 2004): graph-based ranking model selecting key sentences based on keyword extraction. We set 1e to 3 for benchmarking.
Method
R-1
R-2
R-L
HE
Extractive Methods
HMNet (Zhu et al.. 2020): based on trans- former architecture (Vaswani et al., 2017), hicrarchical structure that accommodates long meeting transcripts.
Lead-K
10.2
8.7
8.8
2.6
TextRank
13.4
10.4
8.7
3.0
Fine-tuning Abstructive Methods
HMNet Summ V
17.6
13.4
15.5
20.4
15.4
18.5
3.6
SUMM (Zhang al., 2022): multi-stage framework that adapts coarse-to- ap- proach and specialized handling long input.
DIALOGLM
19.5
10.5
14.4
3.5
Direct Prompting (GPT-3.5) Methods
Divide-and-Summarize
40.6
8.3
19.2
3.8
A.C.E. Divide-and-Summariz
41.1
8.0
16.9
DIALOGLM (Zhong et al., 2022): a pre- trained model that features a window-based de-noising approach. The model also leverage combining sparse attention and conventional attention to process long input.
Iterative Revision
40.4
8.6
17.8
4.3
Table 6: Abstractive summarization task results ported in ROUGE scores. A.C.E. denotes Accumulative Context Enhanced. HE denotes overall human evalua- tion score (1 to 5 rating scale).
ierii-ann-SSmmmrtin we prompt GPT-3.5 to obtain individual summary of each utter- ance, and then integrate summaries to form complete summary (Koay et al., 2021).
4.3 Task 3: Stance-specific Summarization
An overall summary of argumentative dialogue, such as debate meeting, is time-consuming comprehend for readers who wish directly cap- ture stance-specific information. Motivated by this intuition, we propose an integrated task that com- bines Task and 2.
Accumulative Context Enhanced Divide-and- Summarize: similar to the former method, yet we provide accumulative summary of previ- ous part of the debate additional context.
Iterative Revision: first, prompt the model to obtain the summary of the first utterance, then we provide both previously generated summary and new utterance and ask the LLM to revise the summary based on the new
Task Definition Given debate D {U: (G, &)) and a designated stance so [pro, con). J. Each utterance consists of piece of text oj (utterance) and a stance s;. The task is to (1) produce stance speccfcc subset D.,, = (U: =

---

(G, 94 so)] and (2) generate a stance-specific summary so based Deo
Abstractive (overall)
Pro-specific
Con-specific
50
45
Experiment Setup For the abstractive methods, we apply a pipeline strategy by first fine-tuning the models (HMNet, Summ and DIALOGLM) on stance-specifie utterances and their corresponding stance-specific summaries. Next, we ask GPT-3.5 few-shot prompting (for its best performance in Task to distinguish and create a stance-specific subset out of the test set, and then request the mod- els to summarize the utterances. With respect to direct prompting methods, we add simply an in- struction asking the model to distinguish stances of given utterances and utilize the ones whose stance match the designated stance (see Table 12).
40
35
25
20
10
Summ^N
DnS
A.C.E. DnS
L.R.
Figure 3: Comparison of R-1 scores of four methods on both overall summarization and stance-specific sum- marization tasks. DnS:
A.C.E. Accumalative Context Enhunced; L.R.: llerative Revi- sion.
abstractive fine-tuning methods (HMNet, Summ and DIALOGLM), the stance-specific summariza- tion metrics lower compared with the ones in overall summarization task. This may due a non- perfect accuracy in the previous stance detection step. On the other hand, the performances of di- rect prompting methods are improved (especially Iterative Revision). This shows that a preceding highly accurate stance-detection source text be- fore summarizing could likely to benefit summa- rization tasks argumentative dialogues, and inaccurate one may do the opposite.
Method
Stance
R-1
R-2
R-L
IIE
Pipeline Abstractive Methods
Pro
16.2
10.2
3.6
GPT-3.5 HMNet
Con
15.7
10.5
14.2
3.6
Pro
18.2
14.3
3.8
GPT-3.5 Summ
Con
18.2
13.4
17.2
3.9
Pro
18.2
11.2
15.2
3.7
GPT-3.5 + DialogLM
Con
19.2
10.7
16.3
3.8
Direct Prompting (GPT:3.5) Methods
Pro
41.2
10.1
17.8
4.0
Divide-ad-Summmariz
Con
42.6
11.0
4.2
A.C.E.
Pro
39.0
8.2
18.8
4.4
Conclusions
Con
42.2
10.6
4.3
Pro
44.5
11.2
19.2
4.6
We have presented ORCHID, novel dataset for target-independen stance detection and argumen- tative dialogue summarization in Chinese. Our dataset features in rhetorical real-world debates, multi-domain topics, stance-annotated utterances and stance-specific summaries, which invite fu- ture utilization to progress various NLP tasks. We benchmark several baseline stance detection and summarization methods on our dataset. Further- more, we propose new integrated task that shows potential in improving summarization argumen- tative dialogues. Future work can include, for example, devising models to obtain higher stance-detection accuracy, designing better metrics to better evaluate quality summarization, and perhaps developing methods to scale up the size and further augment this dataset.
Iterative Revision
Con
44.4
11.0
17.5
4.6
Table 7: Stance-specific summarization task results reported in ROUGE scores. A.C.E. denotes Accumu- lative Context Enhanced. HE denotes overall human evaluation score to rating scale).
Results The weak evaluation results, as sum- marised in Table 7, establish that the dataset is very challenging and the proposed stance-specing -seecffc summarization task worthy of future exploration on argumentative texts. End-to-end direct prompt- ing methods are better than the pipeline abstractive methods.
While the comparability between the results from the two summarization tasks should be fur- ther examined, singled the R-1 scores of N (for its higher results than the other two ab- stractive methods) and direct prompting methods in both summarization tasks for closer comparison as shown in Figure 3. We observe that, for the three

---

Limitations
Acknowledgement
We appreciate the constructive discussions and in- sightful comments of all reviewers. We thank Boyu Liu for participating the initial data collection. Spa- cial thanks to Sihan Zhao from Parsons School of Design for her illustrations in Figure
We are aware of some limitations of ORCHID.
Data Bias Admittedly, despite our efforts, po- tential bias may have been introduced by the de- mographics of annotators (see Table 8), and we acknowledge it limitation of the annotation process. Since the selected debates are drawn from inter-university debate competitions across 2014 14 2023, all utterances were made by students in higher education institutions, and most of the de- baters are of East Asian origin.
Ethies Considerations
Annotator and evaluator Information
We completed the manual annotation and human evaluation by the collaborators within our research group. The demographic information of them is shown in Table 8.
Demographic Characteristics
Value
Privacy and Licensing
Total Participants
12
All debate videos we collected were released to public by competition organizers on major video sharing platforms (see Appendix E for details). The ASR system we employed in data construction is the iFLYTEK 2022 real-time ASR package' In rare cases, real names of debaters were called dur- ing a debate. The annotators were requested to anonymize all personal information during the an- notation process.
Age
[24, 46]
Sex (Female/ Male)
4 8
Mandarin Chinese Proficiency
native
Education
undergraduate"
Table 8: The demographic information of the annota- tors. "All evaluators have received at least undergradu- ate level education.
Summary Formation While we argue that the final remarks of the last round of debate match in this dataset serve comprehensive summaries of the whole debate. First of all, is advocated by the feature of debate competitions we collected: a debater from each side expected provide a summative and comprehensive remark that cover key statements and arguments of their team, and the last round of a match is called the 'Concluding Phase' in those competitions. Secondly, compared to the summaries written by annotators, original remarks from professional debaters more con- sistent with the debate contents and strongly stance- based, which are in line with the key stance-specific feature that we wish to highlight in this study.
References
Emily Allaway and Kathleen McKeown. 2020. Zero- Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), P),, pages 8913- 8931, Online. Association for Computational Lin- guistics.
Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed, 2023. A systematic review of machine learning tech- niques for stance detection and its applications. Neu- ral Computing and Applications, 35.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din- uzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. Pro- ceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume Long Papers, pages 251 251-261, Valencia, Spain. Association for Computational Lin- guistics.
Nonetheless taking final remarks as summaries has its limitations: in some cases, unseen infor- mation or improvised inference were added by the debaters who made the closing remarks. Also, con- cluding statements in competitive debate scenario are relatively long, resulting low compression ra- tios. In this dataset, the compression ratios (sum- mary length debate length) are 0.43 and 0.41 for proposition and opposition side (c.g., SAMsum's one is 0.30).
Valentin Barrière, Alexandra Balahur, and Brian Ravenet. 2022. Debating europe: A multilingual multi-target stance classification dataset of online de- bates. In Proceedings of the LREC 2022 workshop
/tttps://global w.f.un.cnn/roductt....
real-time-s

---

on Natural Language Processing for Political Sci- ences, VoliticaNLLPLLBB 2022, Marseille, France, June 20-25, 2022, pages 16-21. European Language Resources Association.
(Volume 1: Long Papers), pages 2277-2287, Dublin, Ireland. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20 2031):37-46
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang. and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL. pages 657-668. Association for Computational Lin- guistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese BERT. JEEE ACM Trans, Audio Speech Lang. Process. 29:3504-3514.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Mclissa Kronenthal, Guillaume Lathoud. Mike Lincoln, Agnes Lisowska. lain McCowan, Wilfried Post, Den- nis Reidsma, and Pierre Wellner. 2005. The AMI meeting corpus: A arc-announcement. In Machine Learning for Multimodal Interaction, Second Interna- tional Workshop, MLMI 2005, Edinburgh, UK, July 11 11-13, 2005, Revised Selected Papers, volume 3869 of Lecture Notes in Computer Science, pages 28-39. Springer.
Christine De Kock and Andreas Vlachos. 2021. beg to differ: A study of constructive disagreement in on- line conversations. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2017-2027, Online. Association for Computational Linguistics.
Leon Derezynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for ru- mours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEvvl-2011), pages 69-76, Vancouver, Canada. Association for Computational Linguistics.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems. ACM SIGKDD Explorations Newsletter, 19(2)255-35.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for abstrac- tive screemplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 8602-8615. Dublin, Ireland. Association for Compu- tational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pages 4171-4186. Association for Computational Linguisties.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Kallison-Burchh and Dan Roth. 2019. Seeing things from a different angle:discovering diverse perspec- tives about claims. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 542-557, Minncapolis, Minnesota. As- sociation for Computational Linguistics.
Essi Durmus and Claire Cardie. 2019. A corpus for modeling user and language effects in argumenta- tion on online debating. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019. Volume 1: Long Pupers, pages 602 607. Association for Computational Linguistics.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. Dialogsum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACI/JCNLP 2021, On- line Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACI. pages 5062-5074. Associ- ation for Computational I. Linguistics.
Alexander R. Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir R. Radev. 2021. Convosumm: Conversation summa- rization benchmark and improved abstractive sum- marization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,
Liying Cheng, Lidong Bing, Ruidan He, Qian Yu, Yan Zhang, and Luo Si. 2022. IAM: A comprehensive and large-scale dataset for integrated argument min- ing tasks. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics

---

Kuan-Hao Huang, Chen Li, and Kai- Sei-eei Chang. 2020. Generating sports news from live commentary: A chinese dataset for sports game summarization. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pages tion for Computational Linguisties.
Associa-
Youmna Farag, Charlotte Brand, Jacopo Amidei, Paul Piwek, Tom Stafford, Svetlana Stoyanchev, and An- dreas Vlachos. 2022. Opening up minds with argu- mentative dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 4569-4582, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Natural Language Processing, AACIJICCNLP 2020. Suzhou, China, December 4-7, 2020, pages Association for Computational Linguistics.
Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg. Andreas Stolcke, and Chuck Wooters. 2003. The ICSI meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP '03, Hong Kong, April 6-10, 2003, pages 364-367. IEBE.
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2022. Msamsum: Towards benchmarking multi-lingual di- alogue summarization. In Proceedings of the Sec- ond DialDoc Workshop on Document- -rroundd-g Dia-
logue and Conversational Question Answering. Di- alDoc@ ACL 2022, Dublin, Ireland, May 26. 2022, pages 1- Association for Computational Linguis- tics.
Misha Khalman, Yao Zhao, and Mohammad Saleh. 2021. ForumSum: A multi-speaket conversation summarization dataset. In Findings of the Associa- rion for Computational Linguistics: EMNLP 2021, pages 4592-4599, Punta Cana, Dominican Republic. Association for Computational Linguisties.
William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceed- ings of the 2016 conference of the North American chapter of the association for computational linguis- fics: Human language technologies. ACL.
Jia Jin Koay, Alexander Roustai, Xiaojin Dai, and Fei Liu. 2021. A sliding-window approach to automatic creation of meeting minutes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Stu- dent Research Workshop, pages 68 75, Online. Asso- ciation for Computational Linguistics.
Daniel Gillick, Korbinian Riedhammer, Benoît Favre, and Dilek Hakkani- Türr. 2009. A global optimization framework for meeting summarization. In Proceed- ings of the IEEE International Conference on Acous- rics, Speech, and Signal Processing, ICASSP 2009, 19-24 April 2009, Taipei, Taiwan, pages 4769-4772. IEEE.
Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Sure. 53(1).
Lakshmi Prasanna Kumar and Arman Kabiri. 2022. Meeting summarization: A survey of the state of the art. CoRR, abs/2212.08206.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. CoRR, abs/1911.12237.
John Lawrence and Chris Reed. 2019. Argument min- ing: A survey. Computational Linguistics, 75(4) 818.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ah- met Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: Ru- mourEval, determining rumour veracity and support for rumours. In Proceedings the 13th International Workshop on Semantic Evaluation, pages 845 854, Minneapolis, Minnesota, USA. Association for Com- putational Linguistics.
Yupeng Li, Haorui He, Shaonan Wang, Francis C. M. Lau, and Yunya Song. 2022. Improved target-specific stance detection on social media platforms by delving into conversation threads.
Chin- Yew Lin. 2004. Rouge: evaluation of summaries. In Text summarization branches out, pages 74-81.
A package for automatic
Momchil Hardalov, Arnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance de- tection for mis- and disinformation identification. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1279-1277, Seattle, United States. Association for Computational Lin- guistics.
Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong, 2021. CSDS: A fine-grained chinese dataset for customer service dialogue summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominicun Republic, 7-11 November, 2021, pages 4436-4451. Association for Computa- tional Linguistics.
Kazi Saidul Hasan and Vincent Ng. 2013. Stance clas- sification of ideological debates: Data, models, fea- tures, and constraints. In Proceedings of the Sixth International Joint Conference Natural Language Processing, pages 3348-1356, Nagoya, Japan. Asian Federation of Natural Language Processing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.

---

Roberta: robustly optimized bert pretraining ap- proach.
2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1194-1212, Seat- tle, United States. Association for Computational Linguistics.
Fabrizio Macagno. 2000. Types of dialogue, dialectical relevance, and textual congruity. Anthropology & Philosophy, 801-2):101-211
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Papers, pages 1073-1083. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004, Textrank: Bring- ing order into text. In Proceedings of the 2004 Con- ference Empirical Methods in Natural Language Processing, EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in con- junction with ACI. 2004, 25-26 July 2004, Barcelona, Spain, pages 404-411. ACL
Guokan Shang. Wensi Ding, Zekun Zhang, Antoine J.-P. Tixier, Polykarpos Meladianos, Michalis Vazirgian- nis, and Jean- Bean-Pierre Lorré. 2018. Unsupervised ab- stractive meeting summarization with multi-sentence compression and budgeted submodular maximiza- tion. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Vol- ume 1: Long Papers, pages 664-674, Association for Computational Linguistics.
Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong- Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, and Noam Slonim. 201 8. recorded debating dataset. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, IRREC 2018, Miyazaki, Japan, May 7-1 2018. European Language Re- sources Association (ELRA).
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) pages 31- 41, San Diego California. Association for Computa- tional Linguistics.
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In Pro- ceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume 2, Short Papers, pages 551-557, Valencia, Spain. Association for Computational Lin- guistics.
Anna Nedoluzhko, Muskaan Singh, Marie Hledíková, Tirthankar Ghosal, and Ondřej Bojar. 2022. ELITR minuting corpus: novel dataset for automatic minuting from multi-party meetings in English and Czech. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, pages 3174- 3182, Marseille, France. European Language Re- sources Association.
Yan Song. Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing medical conversations via identifying important utterances. In Proceedings of the 28th International Conference on Computational Linguis- tics, COLING 2020, Barcelona, Spain (Online), De- cember 8-13, 2020, pages 717-729. International Committee on Computational Linguistics.
Dong Nguyen, A. Seza Dogruöz, Carolyn P. Rosé, and Franciska de Jong. 2016. Survey: Computational so- ciolinguistics: A Survey. Computational Linguistics, 4233):537-553.
Jannis Vamvas and Rico Sennrich. 2020. X-stance: A multilingual multi-target dataset for stance detection.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser. and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
OpenAT. 2022. 16k
Tae Jin Park, Naoyuki Kanda, Dimitrios Dimitri- adis, Kyu Han, Shinji Watanabe, and Shrikanth Narayanan. 2022. A review of speaker diarization: Recent advances with deep learning. Computer Speech & Language, 72:101317.
Douglas Walton. 2008. Informal logic: A pragmatic approach. Cambridge University Press.
H Prakken al. 2020. persuasive chatbot using a crowd-sourced argument graph and concerns. Com- putational Models of Argument, 326:9.
Ke Wang and Xiaojun Wan. 2021. Transsum: Trans- lating aspect and sentiment embeddings for self- supervised opinion summarization. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IICNLP 2021 of Findings of ACL. pages 729-742, Association for Computational Lin- guistics.
Allen Roush and Arvind Balaji. 2020. Debatesum: large-scale argument mining and summarization dataset. CoRR, abs/201 107251.
Abraham Sanders, Tomek Strzalkowski, Mei Si, Al- bert Chang, Deepanshu Dey, Jonas Braasch, and Dakuo Wang. 2022. Towards a progression-aware autonomous dialogue agent. In Proceedings of the
Han Wu, Mingjie Zhan, Haochen Tan, Zhaohui Hou, Ding Liang, and Lingi Song. 2023. Vesum: versa- tile chinese meeting summarization dataset.

---

Ruifeng Xu, Yu Zhou, Dongyin Wu, Lin Gui, Jiachen Du, and Yun Xue. 2016. Overview of nipec shared task 4: Stance detection in chinese microblogs. vol- ume 10102, pages 907-916.
A Details about the Selected Debate Competitions
All of selected debate competitions adopt combi- nation of the British Parliamentary and Australia- Asian debate format with minor variations. Such formats consist four debate phases: opening, rebut- tal, free discussion and closing. A group of four debaters make up one side of debate. Convention- ally, the first speaker gives the opening statement; the second speaker and third speaker participates most in rebuttal phase; all members join free dis- cussion, and the forth speaker delivers closing statement in the end. In all selected debate compe- titions, there are adjudicators who introduce teams, state topics and rules, maintain order, and control the process of debates.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Summit: Iterative text summarization via chatgpt.
Justine Zhang, Ravi Kumar, Sujith Ravi, and Cris- tian Baeecu--iculesuu-MMii 2016. Conversational flow in oxford-style debates. In NAACL HLT 2016, The 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, San Diego Califor- nia, USA, June 12-17, 2016, pages 136-141. The Association for Computational Linguisties.
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed H Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summ" n: multi-stage summarization framework for long input dialogues and documents. In ACL 2022.
Competition-specife variations may include: (I) Changing the participants of some sessions: for example, inquiries in rebuttal phase are made by the forth speaker, but some competitions may ask the second speaker to make those inquiries. (2) Adding 'surprise attack' session: for instance, both side may be given a one-time opportunity launch a surprise attack between two debate phases, by making either an inquiry or statement. In addi- tion, while the main games of most selected debate competitions takes standard formats, they some- times host exhibition or entertainment matches that follow special rules.
Ming Zhong. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialogim: Pre-trained model for long dialogue understanding and summa- rization.
Ming Zhong. Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha. Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905 5921, Online. Association for Computational Linguisties.
B Reproducibility Statement
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 202] Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
Regarding summarization methods we bench- marked, HMNet is puubicly available at https://githun com/microosfft//HNtt
SUMM" is publicly available at https:
www.tt....m..uniiiii. http://coossunngrouuu/uummm and DIALOGLM is publicly available at https: thub com mi wttp///iiiroooo//iloo/oo//oooo/o For L.M--propting methods, we specifically employed the fixed 3pt-3.5-turbo-16kk snapshot model for reproducibility purpose, which is available at docs/ https/mddlssgggtttt 3 5/. We kept parameters de- fault, setting temperature to 0.
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xue- dong Huang. 2020. A hicrarchical network for ab- stractive meeting summarization with cross-domain pretraining. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Yicheng Zou, Lujun Zhao, Yangyang Kang, Jun Lin, Minlong Peng, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021. Topic- Topic-oriented spoken dialogue summariza- tion for customer service with saliency-aware topic modeling. In Shirty-Fifth AAAI Conference on Ar- rificial Intelligence, AAAI 2021, Thirty Third Con- ference on Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages I 14665-14673. AAAI Press.
www.ai..peeiiiccommm com/
Dataset Update
Since we further expanded and updated the dataset after the initial benchmark, statistics are subject to change. Find the up-to-date ORCHID and corresponding statistics at https: //gi httggtttbb.... thub. xiutian/OrChiD

---

Statistics
Value
the generated summaries by aforementioned met- rics. The instructions on both summarization tasks are the same.
# Total debates
1,036
# Total unique topics
405
# Total utterances
12,019
# Total stance-specific summaries
2,072
Avg. debate length
18,107
Avg. # utterances per debate
11.6
Avg. utterances length
1,331
Avg. stance-speeific summary length
1,660
Table 9:
Overview statistics for the 2023-06-15 snap- shot of ORCHiD that was tested against in $4. Dehate, utterance summary average lengths are measured in Chinese characters. "Closing remarks were excluded from calculation.
Labels Explanation
Label
Meaning
PRO
Proposition side
Stance
CON
Opposition side
MIXED
Mixed with both sides
Position
Dehater No. in proposition side Debater No. in proposition side
C$
Discussion between debuter No. # proposition side and debuter No. opposition side.
Debater
P#C$ CSP#
FREF
Free discussion involving dehaters
SUM
Closing statement, summary
SA-ST
Surprise attack by statement
SA-IN
Surprise attack by inquiry
Human Evaluation
Metrics Following Allen Institute for Al's GE- NIE leaderboard we choose four aspects as hu- man evaluation metrics. The evaluators are re- quested to rate generated summaries by four as- pects: 1) conciseness, 2) fluency, 3) faithfulness (non-hallucination) and 4) informativeness, follow- ing a rating scale. An overall score is caleu- lated by averaging scores across the aspects with the same weights.
Instructions We randomly split the test set into 12 groups of debates, each consisting around 8 de- bates. Every evaluator is randomly and exclusively assigned one group of debates. For every debate, we ask the evaluators 1) read the transcript, 2) read the gold reference summarics, 3) read and rate
https:: hwwp///eaabbbrrdd.e/../// l enai, org/genie-xseee submissions ssibmm/uooo//buut

---

E
Data Sources
Competition Year
Competition Name
Organizer Official Release Site
Curated
https://www.yyuuubbe.cmm wttp:/uuwb.......u..ueesssssasaa
2013-2023
Ssia-Pacific Intervarsity Chinese Debate Toumament
4 11 9F-H
2018-2022
321
https://pppce..iibbill...o/oo//
Chinese Debate World Cup
[ 1p MF
2014-2023
539
https://spcee.bblbblllcccm/m///
International Chinese Debating Competition
LL ( 1R 12 MA
https://sppac..biiibiil.
The World Mandarin Debating Championship
H Mi X (1 Si
2022
31 htttp:://space..bi1ttt htt.b/lll..oo.eeeeeeeeeeeeeeeee com/1382996
Stars of the GBA International Chinese Debate Tournament
Table I0: Overview of data sources. All collected debate videos released by the official accounts of debate competition organizers at public accessible video sharing platforms including YouTube and Bilibili.
F
Prompts
Task 1: Stance Detection
Method
Prompt
Deternine the stance of following utierance and outgut PRO (proposition side). CON (opposition side) or MIXED (mixed with both sides). Target Clain: pposition_stateet statement)
7em Shot
Utteranc:: (utterance)
Stance:
Consider following examples:
Target fb0 WANTAEEEEEEEEAAA BRITE El (lllowwing public power fale undernining press freedom, ") Utterance: A A (UR lil H 1000) FSCt A 3 RP, HJ 12 ('
(We believe that there is general risk in the incentive and operation of public power restricting fake news, so interfering with journalists and their published content will destroy the freedom of the press.*)
Target Clain: $ - Me Fliinnee monetary policy does more larm than good for curo countries ")) aud fr 1i
Ulterance: RF # 7 7 LE ig
(Only by introducing competition can promote industrics in Europe make reasonable adjustments, and let advantagcous industrics replace weak industries, so that every country Europe can develop better.")
Few
Stance: CON
Targer Claim:
The masses should demand that political leaders become moral models. ")
Unterance: #t FE A- FF1119900 Aniti te 2t ti 18 [C? (1 woold like to ask there is any difference
hetweee your
moral person model.")
Stance: MIXED
Determine the stance following utterance and output PRO (proposition side), CON (opposition side) or MIXED (mixed with both sides). possiiion Ulterance: [utlerance]
Target [position_stateement) statement]
Stance:
Table 11: Stance detection task prompts for LLM prompting methods. *Attaa prompts do not include English translated text.

---

Task 2: Abstractive Summarization
Method
Prompt
Divide
You are tasked to summarize an utterance from formal debate.
-and
All your responses should be in Chinese. Utterance: (ullerance)
-Summarize
A.C.E.
You are tasked to summarize an utterance from formal debate.
Divide
Here is also summary of the previous part of the debate for your reference.
and
All your responses should be in Chinese. Utterance: (utterance)
-Summarize
Here is summary
previous part formal debate. You are tasked to revise the summary utilizing the following utterances.
Tterative Revision
All your responses should be in Chinese. Utterance: (ulterance)
Table 12: Abstractive summarization task prompts for -Lsseaded methods.
Task 3: Stance-specific Summarization
Method
Prompt
are
First deternine the stanee of the following utterance and ourpot PRO (proposition side), CON (opposition side) MIXED (Mixed with both sides). the stance matches [slunce], summurize from the perspcttive of the (siance) side and return the summarr.. matched [stanco], "skipping". All your responses should be in Chinese. Utteranee: [utternnce]
Divide
and
the
Summarize
tasked summarize utterance from formal dehate. First determine the stanee of the following utterance and output PRO (propesition side), CON (opposition side) MIXED (Mixed with both sides). the matches [stance] sammarize
A.C.E. Divide
the perspactive
(stance) side summary.
the stance does matiched [stance), return "skipping".
Here is also summary of the previous part of the dehate for your reference.
-Smmmrize
All your responses should Chinese. Utterrnce: [utterance)
Here is the summary of the previous part of formal dehate.
summary utilizing following utterances.
First determine the stance of the following utterance and output PRO (proposition side). CON (opposition side) MIXED (Mixed with both sides). the stance matches [stance], summarize from the perspective of the (stance) side and return the revised summary. change the summary. All your responses should he in Chinese. Utterance: [utterance]
Revision
matched (stanco),
Table 13: Stance-specific summarization task prompts for LLM-bbsse methods.

---

G
A Detailed Example of the Dataset
Topic
Whether technology is ethically neutral?
Pro Statement
(Techoooog ethically neutral. *)
Con Statement
t 17 (TTehhooloy not ethically neutral. *)
Stance
Debater
Statement
1o t Exploring whether technology is ethically neutral requires looking at whether the technology has a bias and whether such bias lead clear differential outcomes, [...]
B " [...]
PRO
P1
-
SLA #1 I...
MIXED
PIC4
there any difference between having no position taking neutral position?
We feel that having no position and not favoring either side is manifestation of neutrality, [..]]
[ - [-.]]
CON
C1
is clear everyone here that technology. an unconscious non-living entity, cannot choose to be neutral or biased. [...]
# A AA [...]
MIXED
P4C1
In other words, good technology is used by good people and had technology is used by bad people. [...]
[..]]
PRO
P2
(Fng. translation) If had a knife in my hand right now, would it be a good or evil one? The knife can he used kill people. but doctors also it save people, how do you evaluate it? [...]
Bi E (.C. EUS > [. HR St T :2 / IF I. # T A - [...] So you say Heaven and Earth not benevolent, treating all things cattle and dogs, and Heaven and Earth don't even know what joys and sorrows are happening in this world. [..]]
MIXED
P2C3
(T 10lt bliAAA DE fi H 900 [..]]
CON
C2
Whenever technology is created, human values and preferences placed above it, which are ultimately informative and suggestive [...]
7F.2.
f1 [...]
MIXED
P3C2
But wouldn't sharp knife kill faster, so is still a good quality? Killing is not good under your evaluation, but sharpness of the knife is good. [...]
CON
C3
[..]
PRO
P3
[.J]
MIXED
FREE
[..]
( 12 शा E
A 74
mi
W [...]
PRO
SUM
Technology is a two-sided coin. When we talk discuss technology. it is people's gond and evil that give the technology ethical values, not technology itself. Therefore, we should respect technology and should not judge it by our personal likes and dislikes. [...]
L i T S 1 B F3R.
+ [...]
CON
SUM
Today, opponents" debater asked us if we had to give everything stance and definition. Here today, we bravely affirm that is indeed so. [...]
Table 14: An example of ORCHID dataset. One entry consists of 1) debate topic, 2) position statements from both sides, 3) utterances labelled with stance and debater, reference specifi--uumarres of both side. The labels PRO and CON indicate proposition and opposition stance respectively, while SUM denotes summaries. 1) Statements were marked with PRO-P# or CON-C# denoting debater No.# proposition or opposition respectively. 2) Discussion rounds involving both side were labelled MIXED- PECED- indicating the debaters who participated the discussion (see Appendix C for more details on labels). * *Attal data dose not include English translated text.


Reviews:
[{"review_id": "x2HYWku4Dx", "reviewer": "Reviewer_LX9i", "summary": "", "questions": "1) These debate match videos are collected from video-sharing platforms officially released by competition organizers. May list the sources of these videos in the appendix.\n\n2) Can you discuss any potential biases that may have been introduced by having collaborators within your research group complete the manual annotation and how could you avoid these biases?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive feedback! We are glad that you found ORCHID covering these two challenging tasks and enriching existing language source. We appreciate your recognition on our techniques of leveraging features of debates. Below are our responses to specific comments.\n***\n>These debate match videos are collected from video-sharing platforms officially released by competition organizers. May list the sources of these videos in the appendix.\n\nSure! We do maintain a list of such, including competition information and video platform link for each debate we collected. We have added the list in the appendix in our revised version and schedule to provide related information when releasing the data.\n***\n>Can you discuss any potential biases that may have been introduced by having collaborators within your research group complete the manual annotation and how could you avoid these biases?\n\nThank you for pointing out this issue. We were aware of a few possible biases during the annotation process and had taken measures to minimize them.\n\nIn brief, the annotation consists of two major tasks: (1) labelling utterances with stance and debater, and (2) post-editing transcripts to produce reference summaries. We selected 12 persons with sufficient Chinese language skills as the annotators of this study and limited the pool of collaborators to persons whose fields do not overlap with this study to **avoid any conflict of interest**.\n\nWe consider the first task to be **fact-based**, since the truthfulness of labelling can be verified by checking with original videos and hence unanimously agreed upon, which left little room for bias. As described in Section 3.4, we split the 8 annotators (**randomly selected** from the pool of 12) into two groups of four and required a collective label given by each group on each annotation case (labelling stance and debater). We calculated Cohen's K [1] of the labels to evaluate inter-annotator agreement and obtained a **Cohen's K close to 1**, denoting a very high agreement. In fact, the 5 differences the two groups initially disagreed from were reviewed by both groups and reached consent.\n\nRegarding the second task, to avoid any **personal style preference and bias**, two out of 12 annotators were **randomly selected** to double-check the correctness and overall consistency of post-editing, while the remaining 10 annotators were provided with ASR-transcribed text and instructed to generate summaries accordingly. Specifically, as described in Section 3.3, the annotators were requested to rephrase the proposition statements and final remarks of each side into stance-specific summaries. We instructed annotators to **minimize their editing** by limiting changes to removing greetings and shifting addressing to third-person (e.g., from ‘we/I/you’ to ‘the pro/con side’). We have added original instructions in the appendix for the revised version.\n\nAdmittedly, despite our efforts, potential bias may also have been introduced by the **demographics of annotators** (See Table 8), and we acknowledge it as a limitation of the annotation process.\n\n[1] Jacob Cohen. 1960. Coefficient of Agreement for Nominal Scales.\n\n***\n\n>Line 293-294: 'In additional,' -> 'In addition,' or 'Additionally,'.\n\nThank you for catching this typo! We have corrected it in our revised version.\n\n***\nThank you for all the great questions and suggestions! Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper introduces a novel dataset, ORCHID, for debate summarization and target-independent stance detection, which is the first Chinese dataset for these tasks. The paper also proposes a new integrated task, stance-specific summarization, which is suggested by the experiment results to improve the summarization on argumentative dialogues.\n\n2) During the manual annotation step, the authors sufficiently leverage the features of debate competitions, such as (i) using specific signal phrases as round-change indicators and (ii) using the last round concluding remarks from each side as summaries.\n\n3) The paper conducts preliminary experiments, benchmarking classical and newly-suggested tasks on the proposed dataset, covering a wide range of models. The paper provides a versatile testbed for future research.\n\n4) This paper is well-structured and easy to follow.", "reasons_to_reject": "1) The limitations of the dataset (as stated in the Limitations section) and potential biases in the annotation and evaluation may affect the validity and generalizability of the results. However, the authors acknowledge these limitations and suggest future work to address them, which may mitigate these risks.\n", "questions_to_authors": "1) These debate match videos are collected from video-sharing platforms officially released by competition organizers. May list the sources of these videos in the appendix.\n\n2) Can you discuss any potential biases that may have been introduced by having collaborators within your research group complete the manual annotation and how could you avoid these biases?", "paper_topic_and_main_contributions": "This paper proposes the first Chinese dataset for benchmarking both target-independent stance detection and debate summarization, covering 1036 real-world debates over 403 unique topics. This dataset alleviates this language resource shortage in Chinese, and the authors provide the initial benchmark and present a versatile testbed for future research.\n\nThis work focuses on a unique type of dialogue, namely argumentative dialogue, which contains contradictory utterances with conflicting stances and is more complex to be summarized. Further, stance detection in argumentative dialogues is target-independent, where the stances are implicit and should be detected from complete sentences.\n\nBesides these challenges in argumentative dialogue, there still is a lack of multi-domain and annotated Chinese datasets for dialogue (debate) summarization as well as target-independent stance detection. The proposed ORCHID dataset covers these two challenging tasks and enriches the language source for Chinese. The authors also propose a new integrated task, i.e., stance-specific summarization.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 293-294: 'In additional,' -> 'In addition,' or 'Additionally,'."}, "extra_text": {}}, {"review_id": "0nLuxTikZ9", "reviewer": "Reviewer_B1gR", "summary": "", "questions": "[Orchid] Is there any particular reason on why OrChiD is in coloured text?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive comments! We appreciate your recognition of our contribution on introducing a novel dataset and proposing stance-specific summarization task. Below are our responses to specific comments. We number comments and group related ones for cross-reference and readability.\n\n***\n> 1 When providing a novel dataset, an in-depth analysis of collected data in comparison with existing similar datasets is recommended.\n\nIn Table 1 and 2, we have provided comparisons of some key properties of ORCHID with existing related datasets. We have also discussed related datasets and major gaps that motivated us to create ORCHID in \\S Related Work: Existing Datasets.\n\n>1.1 Average number of utterances per dialogue / Average debate length in utterances / Average utterance length\n\nWe have provided an **overview statistics** that include total number of utterances, sentences and debates of ORCHID in **Table 3**. A typical debate match consists of multiple monologue phases and discussion (or ‘rebuttal’) phases. An utterance (used interchangeably with ‘statement’ in this paper) in a debate competition refers to a piece of time-constrained speech that contains multiple sentences, uttered by either one side (monologue phases of a debate labelled by ‘pro’ or ‘con’) or both sides (discussion phases of a debate labelled by ‘mixed’). The time length of an utterance is determined by competition procedure and monitored by debate adjudicators. The **average number of utterans per dialogue (also the **average debate length in utterances** in our case) is **13.60**, while the **average utterance length** is **1331.26** in Chinese characters.\n\nWe have augmented the overview statistics part in our revised version to be more in line with your comments.\n\n>1.2 Definition of argumentative dialogue (is it the same of other datasets? Is it grounded on some argumentation theory like [1, 2]?) \n\nThank you for this informative comment. Our definition of argumentative dialogue is in line with some previous related work and datasets [1,2,3,4,5]. While argumentation theory does not lie in the primary scope of this paper, we will incorporate the references you mentioned and actively consider exploring this with ORCHID in future work.\n\n[1] John Lawrence and Chris Reed. 2019. Argument Mining: A Survey.  \n[2] Farag et al. 2022. Opening up Minds with Argumentative Dialogues.  \n[3] Cheng et al. 2022. IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks.  \n[4] Zong et al. 2021. ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining.  \n[5] Durmus et al. 2019. A Corpus for Modeling User and Language Effects in Argumentation on Online Debating.\n\n>1.3 Dialogue formulation (is the same of other datasets?)\n\nThe dialogue formulation in our dataset is in accord with formal debates. To the best of our knowledge, there currently is no such corpus existing in Chinese. Please kindly find an example of our debates in Table 9.\n\n>1.4 Frequency of mixed-labelled utterances (is the 'mixed' label a property of this particular competitive debate instance?\n\nAs pointed out in [Line 385], the ‘Mixed’ label utterances take **38%** of total utterances across 1036 debates. As compared to other stance detection datasets in Table 2, to the best of our knowledge, ORCHID is the sole dataset with the ‘Mixed’ label.\n\n***\n\n>2 [Introduction, Lines 057-072] It is unclear to what extent the challenges described in this paragraph are reflected in the ORCHID dataset. Could the authors elaborate more in detail?\n\nThe debates in ORCHID are naturally abundant of *contradictory utterances with conflicting stances*. Consequently, we proposed the stance-specific summarization task to reflect this challenge. The next challenge with respect to stance detection is reflected by that the targets of stance detection in ORCHID are complete claims, making the task to be a *target-independent* one. The other challenge, oral features and noises, while not a primary focus of this dataset, can be investigated in future studies since we will provide the audio tracks from collected videos along with raw and cleaned transcripts, as demonstrated in Table 10.\n\n\n***\n\n>3 Does the code include the scripts to collect/process the dataset? Does the data also include audio recordings to, for instance, work with audio modality?\n\nIn fact, we are more than happy to share the dataset with research community! As we stated in the abstract [Line 24] and introduction [Line 125], we will **release all data and code to public after the anonymity period**, in accord with the anonymity policy of EMNLP. The dataset repo links will be added in the revised version.\n\nRegarding details about files and codes to be released, as shown in Appendix B Table 10, the data files we plan to release include **raw audio recording files, ASR-transcribed text, manual-cleaned text and annotated text**. **Data processing scripts** will be provided along with the data.\n***\n\n>4 From which platforms where the original debate videos taken? Which criteria was used to select videos? Which ASR system was used? \n\nThe platforms that we harvested from include three major video sharing platforms in China: Tencent Video, bilibili and Youku; we also taken some videos from YouTube. We employed the 2022 version of the ASR system developed by iFLYTEK under a commercial-use license, as we indicated in [Privacy and Licensing].\n\n***\n\n>5 How many annotators were used, how were these annotators selected and which instructions did they follow (Section 3.2)? How were the eight categories chosen (Section 3.3)? Was the preliminary segmentation of transcripts done manually (Section 3.3)?\n\nAs shown in Table 8, a total of *twelve* annotators were employed in this study. we selected persons with sufficient Chinese language skills and limited the pool of collaborators to persons whose fields do not overlap with this study to *avoid any conflict of interest*. Please see our response to Comment 6 for more annotation details.\n\nThe eight categories were chosen by two steps. Firstly, we brainstormed ten common topic categories based on an overview of the dataset. Then, we mapped each debate to one or more categories, and added new ones if a debate did not fall into any initial category. A total of 18 categories were created. We combined related categories and finalized the number to eight for a balance between informativeness and simplicity.\n\nThe preliminary segmentation was done automatically and manually checked. As we discussed in Section 3 [Lines 311-321], we leverage specific signal phrases given by debate adjudicators. We programmed a script to detect those phrases, separate and label debate sections accordingly. Our annotators then manually checked and corrected mislabeled or wrongly-segmented text.\n\n***\n\n>6 [Summaries, Section 3.3, Lines 322-334] It is unclear how the short and overall summary is created alongside stance-specific ones. Could the authors clarify this aspect?\n\nAs we stated in Section 3.3, we constructed reference summaries on two levels of granularity, and three summaries (pro-specific, con-specific, overall) were created on each level. \n\nThe short stance-specific summaries were created by directly adapting *proposition statements*. Taking the excerpt demonstrated in Table 9 for instance, the pro side proposition statement is ‘technology is ethically neutral’, so the short pro-specific summary is ‘The speaker argues that [pro side proposition statement]’. The short overall summaries are created by concatenating proposition statements from both side: ‘The pro side argues that [pro side proposition statement], and the con side argues that [con side proposition statement].’\n\nThe long summaries were derived from the *final remarks* of debates. At the last round of a formal debate, a debater from each side is expected to provide a thematic and comprehensive remark that cover key statements and arguments of their team. As we stated in Section 3.3, we asked our annotators to manually post-edit them with minimal changes to the contents. To be more specific, in order to control quality and avoid any personal style preference and bias, two out of twelve annotators were *randomly selected* to double-check the correctness and overall consistency of post-editing, while the remaining ten annotators were provided with debate transcripts and instructed to edit the final remarks of each side into stance-specific summaries (the original instructions have been added in the appendix in the revised version). We instructed annotators to **minimize their editing**, limiting to removing greetings and changing addressing to third-person (e.g., from ‘we/I/you’ to ‘the pro/con side’). The long overall summaries are created by concatenating stance-specific summaries from both sides: ‘The pro side argues that [pro-specific summary]. The con side argues that [con side summary]’.\n\n***\n\n>7 [Quality Control, Section 3.4, Lines 347-355] It is unclear why two groups of annotators are created and if the two groups annotate different data. Could the authors further clarify this aspect? Is the IAA computed on all dataset samples? I would also recommend reporting other IAA metrics like Krippendorff's alpha.\n\nWe consider the stance and debate annotation to be **fact-based**, since the truthfulness of labelling can be verified by checking with original videos and hence unanimously agreed upon. The purpose of creating two groups of annotators is to **minimize labelling errors**, and two groups annotated the same whole set of data. As described in Section 3.4, we split eight annotators into two groups of four and required a collective label given by each group on each annotation case (labelling stance and debater). In fact, the 5 differences the two groups initially disagreed from, and both groups reached consent after a collaborative review. We chose Cohen's K because the labels were given by two collective annotators. With just 5 differences present across 14,091 annotation instances, the annotation data yields a **Krippendorff's alpha close to 1** as well.\n\n\n***\n\n>8 [Experiments] How were the models trained? A single train/dev/test split is not a robust evaluation method, especially with a small-to-medium sized dataset like ORCHID. Moreover, are the reported results an average over multiple seed runs? I would recommend a cross-validation routine.\n\nFor some of the methods presented in our paper, no training is required, such as the unsupervised methods Lead-K and TextRank, and the methods based on the OpenAI-provided model, such as Zero-shot & Few-shot GPT-3.5-Turbo Prompting Methods, Divide-and-Summarize, Accumulative Context Enhanced Divide-and-Summarize, and Iterative Revision. For other methods, we trained our models on the train set, adjusted hyperparameters using the validation set, and ran multiple seeds on the test set 3 times, reporting the average results.\n\nOur intention is to provide a set of widely adapted benchmark methods on a unified test set for fair comparison in future research. We appreciate your suggestion for cross-validation and will supplement our results with this approach in the appendix.\n\n***\n\n>9 [Results, Section 4.1, Lines 411-422] What do the authors mean with 'baseline method'? Besides, at Lines 416-417 BERT and RoBERTa are distinguished from LLM-based models, while this is not the case at Lines 418-420.\n\nWe thank the reviewer for pointing out this issue. What we mean ‘baseline method’ here is actually the ‘zero-shot prompting’. The original expression in Lines 417-422 should be corrected as ‘The performances of direct LLM (GPT-3.5 Turbo in our experiment) prompting methods, both zero-shot and few-zero ones, yield substantial improvements compared to transformer-based fine-tuning methods including BERT and RoBERTa’. We have rewritten this part to clarify in our revised manuscript.\n\n***\n\n>10 [Task Definition, Section 4.3, Lines 504-506] It is unclear whether the whole dialogue is needed to produce a stance-specific summary. Could the authors further clarify how each gold stance-specific summary is defined by debaters?\n\nWe argue that the whole dialogue is needed to produce a stance-specific summary. As stated in Section 3.3 and further elaborated in our response to Comment 6, since the gold summaries are adapted from the remarks given by the debaters who had received the whole previous debate contents, it is only fair to define the input of this task the whole dialogue (excluding their final remarks). Please find the detailed process creating reference summaries in our response to Comment 6.\n\n***\n\n>11 [Section 4.3, Lines 520-521] Could the authors further clarify on the choice of Summ^N? Its performance is slightly better than HMNet and on par with DialogLM.\n\nOur experiment aims to test a versatile set of typical methods of dialogue summarization. Summ^N is chosen because it is a newly-introduced (2022) multi-stage framework for input texts that are longer than the maximum context length of typical pretrained LMs. *Specialization in handling long input* makes it suitable for testing on ORCHID (average debate length of 18,107).\n\n***\n\n>12 [Results, Section 4.3, Lines 517-537] It is unclear whether the results of Section 4.2 and Section 4.3 are comparable (Lines 522-528). To the best of my understanding the summaries in the two sections are quite different. Additionally, it is unclear whether the presence of a stance-specific summary is specific to the selected competitive debate setting or not. The claim at Lines 535-537 is vague and potentially in contrast with the claim at Lines 491-496.\n\nWe thank the reviewer for pointing out this issue. Our intuition for comparing the results between the two tasks to suggest a possibility that, combining stance-specific summaries from both sides may form a better abstractive summary than the one generated without stance classification. We realize it is more comparable that we compare the abstractive summaries generated in Section 4.2 and the combined stance-specific summaries in Section 4.3 with the reference overall summaries. Therefore, we have supplemented the experiment accordingly.\n\nWe agree with you that it is yet untested that whether stance-specific summarization is specific to the selected competitive debate setting or not. However, we believe that the proposed task could be generalized to any argumentative text settings where contradictory stances are present and intent to explore this in future work. The primary focus of this work is to provide a versatile testbed for stance detection and dialogue summarization.\n\nWhat we mean at Lines 491-496 is that although direct LLM prompting methods yielded better results than transformer-based training methods, there is still room for improvement, and our dataset could serve as a testbed for future research improving summarization on argumentative dialogue. We advocate the stance-specific summarization task at Line 535-537. We have rephrased our expressions in our revised version.\n\n***\n\n> 13 Typos and Writing\n\nThank you for catching those typos and unclear expressions! We have carefully corrected them. Specifically, regarding [Section 3.2, Lines 286-289] and [Section 3.3, Lines 318-321], you are correct; the manual correction after the automatic was done by the annotators, and we will clarify our expressions in the revised version. In addition, we have added a reference to Table 3.\n\n***\n\n> 14 [ORCHID] Is there any particular reason on why ORCHID is in coloured text?\n\nYes! We applied a color scheme (red for pro, blue for con, purple for mixed) on dataset name and Figure 1 to highlight the stance-specific feature of ORCHID. However, we would certainly comply with conference policy if there was any restriction on colored text.\n\n***\n\nThank you for all the great questions and suggestions! Hopefully this provides sufficient clarification on our original paper! Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A novel dataset for stance detection and dialogue summarization in Chinese.\n\n- A unique dataset for stance-specific dialogue summarization", "reasons_to_reject": "- Lack of dataset analysis and comparison with existing corpora.\n\n- Unclear data collection process\n\n## Recommendation\n\nThe proposed dataset is a sound contribution: the domain of competitive debates is a suitable candidate for argumentative dialogues and the newly proposed task of stance-specific dialogue summarization is a valuable resource for dialogue analysis. These contributions are further strengthen by the lack of such resources in Chinese.\nNonetheless, I think there are some flaws that should be addressed. First, the lack of a in-depth analysis of the proposed dataset along with comparisons with existing datasets weakens the main contribution of the paper. Second, the data collection process lacks some details, leading to poor reproducibility. Third, many claims regarding the observed experimental results are rather unclear. Please, see `Comments` for more details. Overall, it is hard to place OrChiD with existing literature except for the different domain and the added stance-specific summarization task.\nFor this reason, I consider the paper in its current status not suitable for acceptance.\n\n## Comments\n\n[Lack of dataset analysis] When providing a novel dataset, a in-depth analysis of collected data in comparison with existing similar datasets is recommended. In this work, such analysis is missing, making hard to understand the properties of Orchid. \nA non-exhaustive list includes:\n\t- Average number of utterances per dialogue\n\t- Definition of argumentative dialogue (is it the same of other datasets? Is it grounded on some argumentation theory like [1, 2]?)\n\t- Dialogue formulation (is is the same of other datasets?)\n\t- Frequency of mixed-labelled utterances (is the 'mixed' label a property of this particular competitive debate instance?)\n\t- Average debate length in utterances\n\t- Average utterance length\n\n[Introduction, Lines 057-072] It is unclear to what extent the challenges described in this paragraph are reflected in the Orchid dataset. Could the authors elaborate more in detail?\n\n[Unavailable data and code] I personally find the unavailability of data and code a strong negative point when evaluating a paper. However, I understand that it may not represent a reason to reject a paper. Does the code include the scripts to collect/process the dataset? Does the data also include audio recordings to, for instance, work with audio modality?\n\n[Data Collection] Several details are missing from the description. First, from which platforms where the original debate videos taken? Which criteria was used to select videos? Which ASR system was used? How many annotators were used, how were these annotators selected and which instructions did they follow (Section 3.2)? How were the eight categories chosen (Section 3.3)? Was the preliminary segmentation of transcripts done manually (Section 3.3)? \n\n[Summaries, Section 3.3, Lines 322-334] It is unclear how the short and overall summary is created alongside stance-specific ones. Could the authors clarify this aspect?\n\n[Quality Control, Section 3.4, Lines 347-355] It is unclear why two groups of annotators are created and if the two groups annotate different data. Could the authors further clarify this aspect? Is the IAA computed on all dataset samples? I would also recommend reporting other IAA metrics like Krippendorff's alpha.\n\n[Experiments] How were the models trained? A single train/dev/test split is not a robust evaluation method, especially with a small-to-medium sized dataset like OrChiD. Moreover, are the reported results an average over multiple seed runs? I would recommend a cross-validation routine.\n\n[Results, Section 4.1, Lines 411-422] I'm not sure I understood the paragraph. What do the authors mean with 'baseline method'? Besides, at Lines 416-417 BERT and RoBERTa are distinguished from LLM-based models, while this is not the case at Lines 418-420.\n\n[Task Definition, Section 4.3, Lines 504-506] It is unclear whether the whole dialogue is needed to produce a stance-specific summary. Could the authors further clarify how each gold stance-specific summary is defined by debaters?\n\n[Section 4.3, Lines 520-521] Could the authors further clarify on the choice of Summ^N? Its performance is slightly better than HMNet and on par with DialogLM.\n\n[Results, Section 4.3, Lines 517-537] It is unclear whether the results of Section 4.2 and Section 4.3 are comparable (Lines 522-528). To the best of my understanding the summaries in the two sections are quite different. Additionally, it is unclear whether the presence of a stance-specific summary is specific to the selected competitive debate setting or not. The claim at Lines 535-537 is vague and potentially in contrast with the claim at Lines 491-496.\n\n## Typos and Writing\n\n[Abstract, Line 020] 'propose an integrate summarization task' -> 'propose an integrated summarization task'\n\n[Introduction, Lines 086-087] 'most existing stance detection are' -> 'most existing stance detection datasets are'\n\n[Related Work, Lines 130-131] 'we observe a major amount imbalance'\n\n[Section 2.2, Lines 200-201] 'summarizaiton' -> 'summarization'\n\n[Section 3.1, Lines 267-268] 'see 3.5' -> 'see 3.4'\n\n[Section 3.2, Lines 286-289] I suppose that this task is performed by the hired annotators and not the authors.\n\n[Section 3.3, Lines 318-321] I suppose that this task is performed by the hired annotators and not the authors.\n\n[Section 3.3, Line 323] I would remove 'aforementioned' to make the paragraph self-contained.\n\n[Table 3] Is not referenced in any part of the paper.\n\n[Table 3] '# Total Uniuqe Topics' -> '# Total Unique Topics'\n\n[Section 3.4, Line 354] I would replace 'suggesting' with 'denoting' since a Cohen's K close to 1 is almost perfect agreement.\n\n[Section 4.2, Line 459] 'that adpats a corase-to-fine' -> 'that adapts a coarse-to-fine'\n\n[Figure 3] I would remove 'w/, w/o denotes `with` and `without`'\n\n## References\n\n[1] Douglas Walton. 2008. Informal Logic: A Pragmatic Approach, 2 edition. Cambridge University Press.\n\n[2] Douglas Walton and Fabrizio Macagno. 2007. Types of dialogue, dialectical relevance and textual congruity. Anthropology and Philosophy, 8(1-2):101–120.", "questions_to_authors": "[Orchid] Is there any particular reason on why OrChiD is in coloured text?", "paper_topic_and_main_contributions": "The authors propose OrChiD, a novel dataset on argumentative dialogues in Chinese language. In particular, the dataset is proposed to address three tasks: (i) target-independent stance detection; (ii) debate summarization; and (iii) stance-specific summarization. The latter task is one the main contributions of the paper. Experiments on the three tasks with several models (BERT, RoBERTa and GPT-3.5) show that stance detection is the easier task while there is ample room of improvement for summarization tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PeAJStIV9w", "reviewer": "Reviewer_rcmE", "summary": "", "questions": "1.\tWhat is the difference between target-specific and target-independent stance detection datasets? Authors should explain it more clearly in Section 2.1.\n2.\tHow is the stance-specific summary be obtained in detail? Could the last round of the debate fully represent the summary of the whole debate?\n3.\tWhy do you need to annotate the stance of each utterance since each utterance has related debater annotation and the stance could be simply known from the debater’s team.\n4.\tIn Line 417-422, why do you say that all methods yield substantial improvements compared to the baseline method, what does the baseline method represent here?\n5.\tWhy does one statement has two related debater? (such as the second statement in Table 9, the debater is P1C4).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments! We are glad that you found ORCHID promising and our experiments elaborate. Below are our responses to specific comments.\n\n***\n\n>What is the difference between target-specific and target-independent stance detection datasets? \n\nStance detection aims to classify the stance (‘favor’, ‘against’ and ‘none’) of a piece of comment with respect to a target entity or claim. As we discussed in [Lines 66-72], conventionally the targets of *target-specific* stance detection are **explicit entities** (e.g., ‘Metaverse’), while the targets of *target-independent* stance detection are **claims in the form of complete sentences** (e.g., ‘The commercial value of metaverse is overestimated’) [1, 2, 3]. In our case, three elements in ‘target – comment – stance’ triplet in stance detection task are ‘debate topic – utterance – stance’ correspondingly. The debate topics, or proposition statements are complete sentences, making the dataset to be a *target-independent* one.\n\n[1] Ferreira and Vlachos. 2016. Emergent: A Novel Data-set for Stance Classification. ACL 2016.  \n[2] Küçük et al. 2020. Stance Detection: A Survey.  \n[3] Hardalov et al. 2022. A Survey on Stance Detection for Mis- and Disinformation Identification. NAACL 2022.\n***\n\n>How is the stance-specific summary be obtained in detail? \n>Some of the information is not presented clearly, such as the criteria of modifying the last round debate to obtain the summary. Besides the quality control of summary is also not introduced in the paper.\n\nAs briefed in Section 3.3, we constructed reference summaries on two levels of granularity, and three summaries (pro-specific, con-specific, overall) were created on each level. \n\nThe short stance-specific summaries were created by directly adapting *proposition statements*. Taking the excerpt demonstrated in Table 9 for instance, the pro side proposition statement is ‘technology is ethically neutral’, so the short pro-specific summary is ‘The speaker argues that [pro side proposition statement]’. The short overall summaries are created by concatenating proposition statements from both side: ‘The pro side argues that [pro side proposition statement], and the con side argues that [con side proposition statement].’\n\nThe long summaries were derived from the *final remarks* of debates. At the last round of a formal debate, a debater from each side is expected to provide a thematic and comprehensive remark that cover key statements and arguments of their team. As we stated in Section 3.3, we asked our annotators to manually post-edit them with minimal changes to the contents. To be more specific, in order to control quality and avoid any personal style preference and bias, two out of twelve annotators were *randomly selected* to double-check the correctness and overall consistency of post-editing, while the remaining ten annotators were provided with debate transcripts and instructed to edit the final remarks of each side into stance-specific summaries (the original instructions have been added in the appendix in the revised version). We instructed annotators to **minimize their editing**, limiting to removing greetings and changing addressing to third-person (e.g., from ‘we/I/you’ to ‘the pro/con side’). The long overall summaries are created by concatenating stance-specific summaries from both sides: ‘The pro side argues that [pro-specific summary]. The con side argues that [con side summary]’.\n\n***\n\n>Could the last round of the debate fully represent the summary of the whole debate?\n\nWe argue that the final remarks of the last round of a debate match in this dataset can serve as comprehensive summaries of the whole debate. First of all, as we stated in [Lines 327-330], it is advocated by the feature of debate competitions we collected: a debater from each side is expected to provide a thematic and comprehensive remark that cover key statements and arguments of their team, and the last round of a match is called the *‘Concluding Phase’* in those competitions. Secondly, compared to the summaries written by annotators, original remarks from professional debaters are more consistent with the debate contents and strongly stance-based, which are in line with the key *stance-specific* feature that we wish to highlight in this study.\n\nHowever, as we discussed in [Limitations, Line 567-576], we were aware of some limitations of creating summaries by adapting final remarks. First of all, in some cases, debaters may add new information or inference that is not presented in previous debate phases, making the concluding remarks not entirely ‘faithful’ to the previous contents. In addition, final remarks in competitive debate scenario are relatively long, resulting low compression ratios. The average compression ratios (summary length / debate length) of debates curated in ORCHID are 0.43 and 0.41 for proposition and opposition side respectively (e.g., SAMsum's compression ratio is 0.30) [1].\n\n[1] Gliwa et al. 2019. SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization.\n***\n\n\n>Why do you need to annotate the stance of each utterance since each utterance has related debater annotation and the stance could be simply known from the debater’s team.\n\nWe agree with you and also consider the stance and debate annotation of utterances to be **fact-based**, since the truthfulness of labelling can be verified by checking with original videos and hence unanimously agreed upon. However, an **automatic extraction from original videos is not accurate nor complete**. To be more specific, as we described in Appendix C, the structure of each debate competition varies. Although we programmed scripts to obtain a preliminary segmentation and labelling of transcripts, a manual annotation is necessary to correct wrongly split or labelled utterances, ensuring a high quality of ORCHID.\n\n***\n\n>Why does one statement have two related debaters? (such as the second statement in Table 9, the debater is P1C4).\n\nA typical debate match consists of multiple monologue phases and discussion (or ‘rebuttal’) phases. An utterance (used interchangeably with ‘statement’ in this paper) in a debate match refers to a piece of time-constrained speech that contains multiple sentences, uttered by either one side (**monologue** phases of debates labelled by ‘pro’ or ‘con’) or both sides (**discussion** phases of debates labelled by ‘mixed’). The average utterance length of this dataset is 1331.26 in Chinese characters.\n\nTaking the second utterance in Table 9 for instance, this is a discussion phase in which the forth debater of con side rebutted while the first debater of pro side defended, hence its debater is labelled ‘P1C4’ and stance labelled ‘Mixed’, as we explained in the caption of Table 9.\n\nAlthough we had considered separating the sentences of discussion phases by pro and con sides, yet we turned down the separation for two reasons. On the one hand, automatic methods of *speaker diarization* [1] yielded unacceptable results (accuracy less than 60%), and a human annotation approach separating speakers at sentence level will be highly labor-intensive. On the other hand, more importantly, we realized that each discussion utterance is a **complete linguistic unit**. Separating and concatenating sentences from one side may yield incoherent and inconsistent contents. Therefore, we believe it is more prudent to keep the integrity of each utterance\n\nWe have rephrased and further clarified the terms used to be more in line with your comment in our revised version.\n\n[1] Park et al. 2021. A Review of Speaker Diarization: Recent Advances with Deep Learning.\n\n***\n\n>In Line 417-422, why do you say that all methods yield substantial improvements compared to the baseline method, what does the baseline method represent here?\n\nWe thank the reviewer for pointing out this issue. What we mean ‘baseline method’ here is actually the ‘zero-shot prompting’. The original expression in Lines 417-422 should be corrected as ‘The performances of direct LLM (GPT-3.5 Turbo in our experiment) prompting methods, both zero-shot and few-zero ones, yield substantial improvements compared to transformer-based fine-tuning methods including BERT and RoBERTa’. We have rewritten this part to clarify in our revised manuscript.\n\n***\n\n> The analysis of experiment results is insufficient, as most of the analysis are focused on comparing ROUGE scores. it would be better to make deeper analysis on the challenges of the provided dataset (e.g., error analysis on existing methods).\n\nWe agree with you that ROUGE scores have a few limitations evaluating summarization tasks, and we have incorporated a more detailed human evaluation analysis to supplement our paper. As we stated in [Human Evaluation, Lines 1002-1009], we conducted a human evaluation on summarization tasks and requested the evaluators to rate generated summaries by: (1) conciseness, (2) fluency, (3) faithfulness (non-hallucination) and (4) informativeness, following a 1 to 5 rating scale. An overall score is calculated by averaging scores across aspects with the same weights. We have elaborated the human evaluation results in those dimensions on existing methods to provide an in-depth analysis on the challenges of ORCHID in our revised version.\n\n***\n\nThank you for all the great questions and suggestions! Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe dataset could greatly push the research on Chinese dialogue summarization and stance detection, and the authors promise to make the dataset available to public.\n2.\tThe experiment is elaborate, including traditional summarization methods, recent long document summarization methods and unsupervised LLM methods. \n3.\tSummary of existing work is detailed and informative. It could let readers quickly understand the existing research progress of stance detection and dialogue summarization.", "reasons_to_reject": "1.\tSome of the information is not presented clearly, such as the criteria of modifying the last round debate to obtain the summary. Besides the quality control of summary is also not introduced in the paper.\n2.\tThe analysis of experiment results is insufficient, as most of the analysis are focused on comparing ROUGE scores. it would be better to make deeper analysis on the challenges of the provided dataset (e.g., error analysis on existing methods).\n3.\tThe authors only introduce a pipeline system to generate stance-specific summary, an end-to-end supervised method may perform better on this specific task.", "questions_to_authors": "1.\tWhat is the difference between target-specific and target-independent stance detection datasets? Authors should explain it more clearly in Section 2.1.\n2.\tHow is the stance-specific summary be obtained in detail? Could the last round of the debate fully represent the summary of the whole debate?\n3.\tWhy do you need to annotate the stance of each utterance since each utterance has related debater annotation and the stance could be simply known from the debater’s team.\n4.\tIn Line 417-422, why do you say that all methods yield substantial improvements compared to the baseline method, what does the baseline method represent here?\n5.\tWhy does one statement has two related debater? (such as the second statement in Table 9, the debater is P1C4).", "paper_topic_and_main_contributions": "This paper provides a new Chinese debate dialogue dataset with stance and summary annotation. Apart from the overall summary for the whole debate, each dialogue in this dataset also has a stance-specific summary. The authors then apply some baseline methods (pretrained model based methods and large language model based methods) to the new dataset, report their results and give suggestions for future work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new Chinese dataset for stance detection, debate summarization, and stance-specific debate summarization for argumentative dialogues. The dataset comes with a set of baseline methods to address the task; They find that summarization performance is relatively low, while stance detection is found to be a relatively easy task.
The reviewers agree that the paper is unique in that it proposes the first Chinese dataset for these tasks, and the modelling experiments provide a good set of baselines on the tasks. One reviewer also highlights the quality of the related work discussion, while another misses a more detailed description of the dataset collection process (this aspect can be addressed easily enough in the final version of the paper, judging from the author rebuttal). The evaluation contains both automatic metrics and human judgments.
Decision: Accept-Main