Title: RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
Benchmark: RICA
Summary: RICA is a benchmark for evaluating language models' ability to make robust commonsense inferences despite linguistic variations and logical perturbations. Starting from first‑order‑logic templates that encode commonsense axioms, the authors automatically crawl ConceptNet and ATOMIC, then apply 24 perturbation types (negation, antonym, paraphrase, etc.). Experiments show that pre-trained language models perform poorly on robust inference tasks even after fine-tuning, highlighting a significant gap between current AI systems and human-level understanding.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language inference (for commonsense knowledge), first-order logic
Phenomenon defined by authors: Yes
Phenomenon definition: Authors formalize each commonsense axiom as a first‑order‑logic implication and deem a model successful only if it solves all logically‑equivalent probes for that axiom
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: To test language models' commonsense reasoning and robustness to linguistic variations, the authors transform first-order logic axioms into multiple syntactically different statements expressing the same inferential relationship, then evaluate models through task-specific probes, considering models successful only if they perform like humans across all variations of the axioms.
Task item definition: Premise + conclusion statement (+ with a [MAST] token) + a pair of novel entity strings
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2600
Dataset size detail: 2600 testsets (joint human‑verified + curated), 8000 training sets (plus optional 100,000 noisy or 257,000 raw datasets)
Dataset size notes: 8000 (train)
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean + standard deviations
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Ablation studies, perturbation analysis, knowledge control
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning