Title: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
Primary area: interpretability and explainable AI
Keywords: Large Language Models, Interpretability, Item Response Theory, Joint Embeddings, Geometric Analysis

Abstract: Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature. We present **JE-IRT**, a geometric item-response framework that embeds both LLMs and questions in a shared space. For question embeddings, the *direction* encodes semantics and the *norm* encodes difficulty, while correctness on each question is determined by the geometric interaction between the model and question embeddings. This geometry replaces a global ranking of LLMs with topical specialization and enables smooth variation across related questions. Building on this framework, our experimental results reveal that out-of-distribution behavior can be explained through directional alignment, and that larger norms consistently indicate harder questions. Moreover, JE-IRT naturally supports generalization: once the space is learned, new LLMs are added by fitting a single embedding. The learned space further reveals an LLM-internal taxonomy that only partially aligns with human-defined subject categories. JE-IRT thus establishes a unified and interpretable geometric lens that connects LLM abilities with the structure of questions, offering a distinctive perspective on model evaluation and generalization.

TL;DR: JE-IRT reveals LLM abilities through geometry, with direction capturing semantics and norm capturing difficulty.