A rigorous understanding of how language models work demands a rigorous theory of their internal mechanisms, but constructing such a theory is no small task.