AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language
  Models
Authors: name: Jaeho Lee, name: Atharv Chowdhary

Recent benchmarks have probed factual consistency and rhetorical robustness
in Large Language Models (LLMs). However, a knowledge gap exists regarding how
directional framing of factually true statements influences model agreement, a
common scenario for LLM users. AssertBench addresses this by sampling
evidence-supported facts from FEVEROUS, a fact verification dataset. For each
(evidence-backed) fact, we construct two framing prompts: one where the user
claims the statement is factually correct, and another where the user claims it
is incorrect. We then record the model's agreement and reasoning. The desired
outcome is that the model asserts itself, maintaining consistent truth
evaluation across both framings, rather than switching its evaluation to agree
with the user. AssertBench isolates framing-induced variability from the
model's underlying factual knowledge by stratifying results based on the
model's accuracy on the same claims when presented neutrally. In doing so, this
benchmark aims to measure an LLM's ability to "stick to its guns" when
presented with contradictory user assertions about the same fact. The complete
source code is available at https://github.com/achowd32/assert-bench.