MultiLogicNMR(er): A Benchmark and Neural-Symbolic Framework for Non-monotonic Reasoning Tasks with Multiple Extensions
Authors: Yeliang Xiu, Yongmei Liu


Reviewer paper summaries: The paper introduces a benchmark and framework, MultiLogicNMR(er), designed to evaluate and improve the non-monotonic reasoning abilities of large language models (LLMs). Non-monotonic reasoning is crucial for scenarios where conclusions can be retracted based on new information, such as in legal reasoning. The study addresses the limitations of existing benchmarks (δ-NLI and LogicNMR) by focusing on reasoning tasks with multiple extensions, which represent sets of plausible conclusions.
Summary of strengths: 1. The MultiLogicNMR dataset provides a robust evaluation framework for non-monotonic reasoning with multiple extensions, a scenario not adequately addressed by previous benchmarks.
2. The neural-symbolic framework significantly enhances the reasoning performance of LLMs, achieving higher accuracy than both prompt-based and fine-tuned approaches.
Summary of weaknesses: 1. No prompting methods for reasoning, such as chain-of-thought, tree-of-thought prompting results are presented. 
2. The automatically synthesized dataset may not fully capture the complexity of real-world natural language, limiting the practical applicability of the benchmark.
Comments, suggestions and typos: 1. Tables in this paper are poorly presented. 
2. Typo:The δ-NLI and LogicNMR proposed in the existing work have only preliminary explored004the non-monotonic reasoning ability of the005pre-trained language models (LMs) in natu-006ral language. -> The δ-NLI and LogicNMR proposed in the existing work have only preliminarily explored the non-monotonic reasoning ability of pre-trained language models (LMs) in natural language.
Overall assessment scores: 3
Metareview score: 3
Metareview: The paper presents a novel benchmark and framework, MultiLogicNMR(er), aimed at enhancing and evaluating the non-monotonic reasoning capabilities of large language models (LLMs). Non-monotonic reasoning, crucial for scenarios where conclusions can change with new information (e.g., legal reasoning), is explored through a dataset with multiple plausible conclusions (extensions). The proposed neural-symbolic framework, MultiLogicNMRer, integrates symbolic reasoning strategies with LLMs to improve their performance on these tasks.
Summary of reasons to publish: 1. The introduction of MultiLogicNMR, a dataset specifically tailored for evaluating non-monotonic reasoning with multiple extensions, fills a gap in current benchmarks.
2. The benchmark provides a comprehensive evaluation framework, including an out-of-domain test set (MultiLogicNMR_OOD), to assess the generalization capabilities of models.
3. The paper is well-grounded in previous research, providing a comprehensive background on non-monotonic reasoning and its relevance to AI and NLP.