Title: MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference
Benchmark: MSciNLI
Summary: This paper introduces MSCINLI, a new dataset comprising 132,320 sentence pairs from five diverse scientific domains to enhance the study of scientific Natural Language Inference (NLI). Baseline models, including fine-tuned and prompted LLMs, reveal the dataset's challenging nature, as well as performance degradation due to domain shifts, highlighting the unique characteristics of each domain. Additionally, employing both scientific NLI datasets in intermediate task transfer learning showcases improvements in downstream scientific tasks.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Natural language inference (semantic relationship between two sentences), scientific domains
Phenomenon defined by authors: Yes
Phenomenon definition: predicting the semantic relation between two sentences extracted from research articles
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: sentence pairs, multiple choice on semantic relation between sentences
Task definition detail: question, prompt, domain, class, difficulty, response correct/score
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 127,320
Dataset metadata available: Yes
Dataset metadata detail: difficulty, domain
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: mean and variance, t-tests
Metric subscores: Yes
Metric subscores detail: difficulty
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: Yes
Human baseline: Yes
Authors discuss validity: No
Taxonomy: General Science / 
New taxonomy: Specific Application / Physical Sciences