RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
  Scenarios
Authors: name: Ruiwen Zhou; name: Wenyue Hua; name: Liangming Pan; name: Sitao Cheng; name: Xiaobao Wu; name: En Yu; name: William Yang Wang
Published: 2024-12-12 06:08:46

Abstract: This paper introduces RuleArena, a novel and challenging benchmark designed
to evaluate the ability of large language models (LLMs) to follow complex,
real-world rules in reasoning. Covering three practical domains -- airline
baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
proficiency in handling intricate natural language instructions that demand
long-context understanding, logical reasoning, and accurate mathematical
computation. Two key attributes distinguish RuleArena from traditional
rule-based reasoning benchmarks: (1) it extends beyond standard first-order
logic representations, and (2) it is grounded in authentic, practical
scenarios, providing insights into the suitability and reliability of LLMs for
real-world applications. Our findings reveal several notable limitations in
LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
becoming confused by similar but distinct regulations, (2) they cannot
consistently perform accurate mathematical computations, even when they
correctly identify the relevant rules, and (3) in general, they perform poorly
in the benchmark. These results highlight significant challenges in advancing
LLMs' rule-guided reasoning capabilities in real-life applications.