Title: Hidden in Order: A Theoretical and Empirical Dissection of Positional Bias in LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: Positional Bias, Large Language Models (LLMs), Representation Similarity

Abstract: Large Language Models (LLMs) demonstrate strong capabilities but remain limited by positional bias defined as the tendency to treat content differently based on where it appears in a sequence. This bias creates subtle but consequential distortions in tasks that demand equal treatment of all inputs, such as multi-document reasoning. Although positional effects have been observed, no unified approach has existed to measure them systematically. We present Xayna, a framework of 11 complementary methods spanning information-theoretic, geometric, and probabilistic perspectives. Xayna evaluates internal representations, attention flows, and output behavior across models including GPT-4o, Gemini 2.5, Llama 3, Claude 3.7, Mixtral 8×22B, and DeepSeek R1. Results show consistent patterns: (i) identical content yields divergent representations with near-zero cosine similarity; (ii) attention flows disproportionately from later to earlier tokens; and (iii) primacy dominates behavior, with position-1 preferences reaching 0.892 in Llama 3. Xayna provides both a toolkit for quantifying positional bias and a foundation for developing position-aware evaluation and mitigation strategies, advancing the pursuit of more reliable and equitable language technologies.