Title: Quantifying Mechanistic Gaps in Algorithmic Reasoning via Neural Compilation
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: graph neural networks, neural algorithmic reasoning, neural compilation, mechanistic interpretability, algorithm learning, reasoning, expressivity, neurosymbolic

Abstract: Neural networks can achieve high prediction accuracy on algorithmic reasoning tasks, yet even effective models fail to faithfully replicate ground-truth mechanisms, despite the fact that the training data contains adequate information to learn the underlying algorithms faithfully. 
We refer to this as the \textit{mechanistic gap}, which we analyze by introducing neural compilation for GNNs, which is a novel technique that analytically encodes source algorithms into network parameters, enabling exact computation and direct comparison with conventionally trained models.
Specifically, we analyze graph attention networks (GATv2), because of their high performance on algorithmic reasoning, mathematical similarity to the transformer architecture, and established use in augmenting transformers for NAR.
Our analysis selects algorithms from the CLRS algorithmic reasoning benchmark: BFS, DFS, and Bellman-Ford, which span effective and algorithmically aligned algorithms.
We quantify faithfulness in two ways: external trace predictions, and internal attention mechanism similarity. 
We demonstrate that there are mechanistic gaps even for algorithmically-aligned parallel algorithms like BFS, which achieve near-perfect accuracy but deviate internally from compiled versions.

TL;DR: We introduce neural compilation for GATv2 and propose two measures of mechanistic faithfulness, which we validate experimentally, showing that even algorithmically aligned and parallel models struggle to learn faithfully.