Title: ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization
Benchmark: ORCHID
Summary: The paper proposes a new debate dataset and benchmark in Chinese. The aim of this dataset is to assess model capabilities in stance detection based on dialogue (pro or con), in addition to summarizing the dialogue.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Ability to summarize dialogues and detect the stance of the debaters on the topic.
Phenomenon defined by authors: Yes
Phenomenon definition: "Stance detection and dialogue summarization are two core tasks of
dialogue agents in application scenarios that involve argumentative dialogues."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: (1) stance detection; (2) abstractive summarization; and (3) stance-specific summarization, a new integrated task that we propose.
Task item definition: Task 1 (Stance Detection): Contains an utterance with the label being  "pro, con, mixed". This is a classification task. 

Task 2 (Abstractive Summarization): A full dialogue D. The task is to summarize it.

Task 3 (Stance-specific Summarization): Similar to task task 2, but with a label for every utterance within the debate either "pro" or "con".
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Stance Detection: 1550. Abstractive Summarization: 104. Stance-specific Summarization: 208.
Dataset size notes: Stance Detection: Validate (1,534) Train (11,005) Abstractive Summarization: Validate (104) Train (828) Stance-specific Summarization: Validate (208) Train (1,656)
Dataset metadata available: Yes
Dataset metadata detail: details about the annotators, average conversation length, average summary length
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization