Title: DiscoSense: Commonsense Reasoning with Discourse Connectives
Benchmark: DISCOSENSE
Summary: This paper introduces DISCOSENSE, a benchmark for commonsense reasoning that focuses on understanding various discourse connectives. The task requires selecting the most plausible sentence ending given a preceding context sentence and a specific discourse connective. The benchmark uses Conditional Adversarial Filtering, an extension of Adversarial Filtering, to generate difficult distractor options. Evaluations demonstrate that state-of-the-art language models find DISCOSENSE challenging, suggesting it's a valuable tool for assessing commonsense reasoning.
Contribution: Key contributions include: (1) Creating the DISCOSENSE benchmark targeting commonsense reasoning specifically through the understanding of 37 discourse connectives. (2) Proposing Conditional Adversarial Filtering (CAF) to generate compelling, hard-to-distinguish negative options. (3) Benchmarking numerous state-of-the-art language models, highlighting the difficulty of the task (significant gap to human performance). (4) Demonstrating the utility of DISCOSENSE for transfer learning by improving performance on the HELLASWAG dataset.  
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Commonsense reasoning; Understanding discourse connectives and relations.
Phenomenon defined by authors: Yes
Phenomenon definition: The capability to perform commonsense inference to determine the most plausible sentence ending given a context sentence and a discourse connective. This requires understanding the specific semantic relationship (e.g., causality, contrast, exemplification) implied by the connective and applying world knowledge to select the most coherent and logical continuation.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: To create a more challenging commonsense benchmark less prone to superficial cues or artifacts, by specifically focusing on the reasoning required by discourse connectives and using adversarial methods (CAF) to generate strong distractors.
Task definition: Given a context sentence and a discourse connective, select the most plausible ending sentence from four options, requiring commonsense reasoning based on the connective's meaning.
Task item definition: An instance consists of a context sentence, one of 37 discourse connectives, and four potential ending sentences. One ending is the ground truth (human-written or verified), and the other three are distractors generated via Conditional Adversarial Filtering.
Task definition detail: Contexts are derived from DISCOVERY and DISCOFUSE datasets. Adversarial distractors are generated using a fine-tuned CTRL model. Human verification filters the final set of examples.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Contexts and original endings sourced from DISCOVERY/DISCOFUSE. Three distractor endings generated for each example using Conditional Adversarial Filtering (CAF) involving a fine-tuned CTRL generator and a RoBERTa discriminator. The resulting examples were filtered through a two-step human verification process.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The task isolates reasoning about discourse connectives in a challenging multiple-choice format created via adversarial generation.
Dataset size: Total: 13,056 examples. Test set: 3,757 examples.
Dataset size notes: Train set: 9,299 examples. No Dev split mentioned.
Dataset metadata available: Yes
Dataset metadata detail: Discourse Connective (one of 37 types).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Response format detail: The model must choose the index corresponding to the most plausible ending sentence out of the four provided options.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Accuracy (percentage of correctly chosen plausible endings).
Metric aggregation: Simple Mean
Metric statistics: Accuracy (%), Standard Deviation, Error Rate (%)
Metric subscores: Yes
Metric subscores detail: Error rate analysed per discourse connective. Ablation study analysing the impact of removing context and/or the connective.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Used CAF to generate hard distractors. Employed a 2-step human verification process for filtering. Final dataset demonstrates a significant human-model performance gap (~30 points). Ablation confirms models utilize both context and connectives.
Taxonomy: Reasoning / Commonsense
New taxonomy: General Capability / Commonsense Reasoning
Procedural notes: Detailed explanation of Conditional Adversarial Filtering (CAF) process. Details on human verifier recruitment, training, and compensation. List of 37 included discourse connectives provided. Training hyperparameters specified. Ethical considerations discussed.
Other notes: The use of discourse connectives as the focal point for a commonsense reasoning benchmark is novel. Conditional Adversarial Filtering is a key methodological contribution for creating challenging distractors. The dataset proves effective both as a challenging benchmark and as a resource for transfer learning to related tasks like HELLASWAG.