Title: ROBUSTLR: A Diagnostic Benchmark for Evaluating Logical Robustness of
Deductive Reasoners
Benchmark: RobustLR
Summary: Deductive reasoning is an important skill that modern language models should possess. However, small logical perturbations of deductive reasoning problems can lead to inconsistent model responses. To test this consistency, the paper introduces RobustLR a benchmark consisting of logical problems ("theories") and variations thereof that should be consistenly answered correctly by models.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: robustness of deductive reasoning against small shifts in logical operators or rephrasing.
Phenomenon defined by authors: Yes
Phenomenon definition: "We consider a deductive reasoner (language model) to be logically robust if the model behavior is consistent across various logical perturbations."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Consistency here can be misinterpreted: The perturbations applied to problems cause different conclusions. Consistency is here defined as being accurate across perturbations, i.e. changing the label when the input changes. This is in contrast to many other works that regard consistency as invariance.
Task definition: The task has 2 levels: The underlying task is conducting deductive reasoning. This is a classification problem: "True", "False" "Unknown". The "meta-task" is being consistent across a set of related problems.
Task item definition: One item in the benchmark is a set: "original problem" + a set of perturbations on the problem. Each problem is a set of facts, rules and deduction.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Response format detail: yes
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: mean of weighted-F1 scores
Metric subscores: Yes
Metric subscores detail: different kinds of perturbations of the problem.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors clearly state limitations due to simple composition of rules used for perturbations and the synthetic toy nature of the dataset. They also validate that humans can achieve good scores on the problems while langauge models dont.
Taxonomy: Reasoning / Logical
Alternate taxonomy: Robustness
New taxonomy: General Capability / Logical Reasoning
Other notes: The synthetic nature of the benchmark is very much limiting the ecological validity of the benchmark for real user interaction, but the authors are very clear and transparent about it. The lack of ecological validity is compensated by internal validity.