In Case You Missed It: ARC 'Challenge' Is Not That Challenging
Authors: name: Łukasz Borchmann
Published: 2024-12-23 18:14:36

Abstract: ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily
due to an evaluation setup that prevents direct comparison of answer choices
rather than inherent complexity. Although some researchers have quietly shifted
to a more appropriate scheme over the last year, the implications of this
change have yet to be widely acknowledged. We highlight this overlooked shift,
show how similar evaluation practices falsely imply reasoning deficits in other
benchmarks, and demonstrate that fairer methods dramatically reduce performance
gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing
so, we reveal how evaluation shapes perceived difficulty and offer guidelines
to ensure that multiple-choice evaluations accurately reflect actual model
capabilities.