Title: Circuits, Features, and Heuristics in Molecular Transformers
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Computational Chemistry, Drug Discovery, Mechanistic Interpretability, Molecular Generation, Transformers, Sparse Autoencoders

Abstract: Transformers generate valid and diverse chemical structures, but little is known about the mechanisms that enable these models to understand the rules of molecular representation. We present a mechanistic analysis of autoregressive transformers trained on drug-like small molecules to reveal the computational structure underlying their chemical reasoning capabilities across multiple levels of abstraction. We identify computational primitives that handle low-level syntactic parsing and more abstract chemical validity constraints. Using sparse autoencoders (SAEs), we extract human-understandable feature dictionaries that correspond to chemically relevant activation patterns and present a scalable methodology for identifying pharmacologically meaningful elements in these representations. We validate our findings on downstream tasks and find that mechanistic insights can translate to predictive performance in various practical settings.