Title: The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

mechanistic interpretability

causal alignment

distributed alignment search

linear representation hypothesis
Abstract: The concept of causal abstraction got recently popularised to demystify the opaque decision-making processes of machine learning models; in short, a neural network can be abstracted as a higher-level algorithm if there exists a function which allows us to map between them. Notably, most interpretability papers implement these maps as linear functions, motivated by the linear representation hypothesis: the idea that features are encoded linearly in a model's representations. However, this linearity constraint is not required by the definition of causal abstraction. In this work, we critically examine the concept of causal abstraction by considering arbitrarily powerful alignment maps. In particular, we prove that under reasonable assumptions, any neural network can be mapped to any algorithm, rendering this unrestricted notion of causal abstraction trivial and uninformative. We complement these theoretical findings with empirical evidence, demonstrating that it is possible to perfectly map models to algorithms even when these models are incapable of solving the actual task; e.g.,
on an experiment using randomly initialised language models, our alignment maps reach 100\% interchange-intervention accuracy on the indirect object identification task. This raises the non-linear representation dilemma: if we lift the linearity constraint imposed to alignment maps in causal abstraction analyses, we are left with no principled way to balance the inherent trade-off between these maps' complexity and accuracy. Together, these results suggest an answer to our title's question: causal abstraction is not enough for mechanistic interpretability, as it becomes vacuous without assumptions about how models encode information. Studying the connection between this information-encoding assumption and causal abstraction should lead to exciting future work.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically examines the framework of causal abstraction for mechanistic interpretability. The authors prove that, under reasonable assumptions, allowing for arbitrarily powerful (non-linear) alignment maps renders the notion of causal abstraction vacuous, as any neural network can be perfectly mapped to any algorithm. This theoretical claim is supported by strong empirical evidence on both synthetic tasks and language models, where near-perfect interchange-intervention accuracy (IIA) is achieved even for randomly initialized models that cannot solve the task. The paper frames this finding as the \"non-linear representation dilemma,\" highlighting the fundamental trade-off between the expressive power of an alignment map and the meaningfulness of the resulting explanation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles a foundational and timely problem in interpretability research. The central theorem, proving the vacuity of unconstrained causal abstraction, is a rigorous and significant contribution.\n2. The empirical results provide compelling support for the theoretical claims. The demonstration that complex alignment maps can find perfect abstractions even in randomly initialized or incapable models is a strong piece of evidence.\n\nWeaknesses:\n1. The paper's conclusion feels somewhat tentative and does not fully embrace the strong implications of its own findings. The authors state an \"intuition\" to support the Linear Representation Hypothesis (LRH) but claim they cannot formalize it. This seems to miss an opportunity to propose more nuanced, falsifiable versions of the LRH as a path forward, as I will detail in the questions.\n\nQUESTIONS:\nIn your conclusion, you state an \"intuition\" to support the linear representation hypothesis (LRH) but acknowledge that you \"cannot make this intuition formal.\" This difficulty may arise because your work seems to make the general LRH intractable from both a falsification and verification standpoint:\n* To falsify the universal LRH, one must demonstrate a feature that is provably non-linear. This requires showing both (i) the failure of linear maps and (ii) the success of a non-linear one. However, (i) might never be finished, and for (ii) your theorem shows that a successful alignment with a powerful nonlinear map is a method-induced artifact, providing no evidence that the network actually represents the feature non-linearly. Thus, a required counterexample can never be convincingly established.\n* Simultaneously, any verification by a linear alignment map is weakened. It can always be challenged as a mere approximation of a more complex, unfalsifiable non-linear ground truth with perfect IIA.\n\nGiven that the universal LRH is rendered practically unprovable and unfalsifiable within this framework, shouldn't the paper's main takeaway be a stronger recommendation to abandon it as a useful scientific hypothesis? Instead of relying on intuition, shouldn't the community be guided to formulate and test a portfolio of more modest, but strictly falsifiable, hypotheses? For example:\n* (a) a local, circuit-specific claim (e.g., \"The IOI circuit in GPT-2 Small uses a linearly decodable representation for the 'ABAB pattern' variable\").\n* (b) An instrumental, Pareto-optimal claim (e.g., \"Within a fixed complexity budget, a linear decoder provides the best explanation on the Pareto front of IIA vs. complexity\").\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines a fundamental limitation in causal abstraction methods for mechanistic interpretability. The authors prove that when alignment maps between neural networks and algorithms are allowed to be arbitrarily complex (non-linear), any neural network can be mapped to any algorithm under reasonable assumptions, making causal abstraction vacuous. They demonstrate this through Theorem 1 and empirical validation showing that complex alignment maps achieve perfect interchange intervention accuracy (IIA) even on randomly initialized models incapable of solving target tasks. The work identifies the \"non-linear representation dilemma\" - the inherent trade-off between alignment map complexity and interpretability validity - and argues that causal abstraction implicitly relies on strong linearity assumptions about neural information encoding.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\ntechnical\n- theoretical contribution is mathematically rigorous with well-stated assumptions that are reasonable for practical neural networks (countable input spaces, input-injectivity, output-surjectivity)\n- proof structure is sound and the empirical validation effectively demonstrates the theoretical predictions across multiple tasks\n- experimental design systematically explores alignment map complexity using identity, linear, and non-linear (RevNet) parameterizations, providing evidence that complex maps can achieve high IIA regardless of model capability.\n\npresentation\n- makes inherently complex mathematical concepts accessible to the NeurIPS audience\n- progressive introduction of concepts from basic causal models to distributed abstractions is well-structured\n-  hierarchical equality and IOI tasks provide concrete illustrations of abstract theoretical points\n- figures effectively communicate the key findings\n\nnovelty and significance\n- focus on alignment map complexity in causal abstraction is novel\n- non-linear representation dilemma provides a framework that could redirect research toward principled constraints for alignment maps.\n\n### Weaknesses\n- key assumptions (particularly input-injectivity) may fail in practice (as the authors admit) - \"identity of first argument\" experimental results also show this → somewhat limits the theoretical result\n- empirical validation limited to relatively simple tasks and specific model architectures\n- for Pythia, single-seed reduces statistical confidence\n\n- math in Section 3 is quite dense and could benefit from more intuitive explanations\n- notation becomes complex and inconsistent in places (particularly the intervention notation in equations 4-6)\n- practical implications for practitioners could be articulated more clearly (what is the concrete guidance?)\n\nsignificance and novelty\n- limited to causal abstraction methods\n- complexity-accuracy trade-off, while formalized in a novel context, echoes similar debates in the probing literature (but actually the authors were fairly thorough in citing the complexity-accuracy trade-off literature from probing)\n\nminor: \n - throughout the paper, in figures and text, I'd advise to use different colors/patterns that are colorblind-friendly - you are using red-green contrast, in general the color scheme doesn't seem coherent to me (but well done on the different shadings in figure 1 to distinguish the categories without color). I would probably swap the red-green for blue-orange (given that you use blue in the plots). Overall this is minor though as in the figures you made sure to distinguish the elements without color.\n - figure captions could better explain what readers should conclude from each result\n - l. 99: The notation \"Δ|Y|-1\" is unclear - clarify this is the probability simplex\n - l. 182: Ensure the bijection notation is consistent with later usage\n - Add more intuitive explanations alongside formal definitions (especially Definitions 1-2, 6-7)\n - maybe include a glossary of key terms in the appendix, \n - maybe include a notation table in the appendix listing all symbols\n - proof sketches in the main text for Theorem 1, even if full proof is in appendix\n - consistent subscript/superscript conventions throughout (some inconsistencies in ω vs ε notation)\n - In the introduction, better motivate why this problem matters for the broader ML community\n\nQUESTIONS:\n1. Can you provide a more systematic analysis of when the key assumptions (particularly input-injectivity and output-surjectivity) hold in practice? What are the concrete implications when these assumptions fail, and how common are such failures across different architectures and tasks?\n    \n2. Given that you've identified the non-linear representation dilemma, what specific recommendations would you provide to practitioners currently using causal abstraction methods? How to choose appropriate alignment map complexity? Relatedly, what are the most promising research directions for addressing the identified limitations?\n    \n3. Do your findings extend to other interpretability methods beyond causal abstraction? Are there fundamental insights here that apply more broadly to the interpretability research program?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the application of causal abstractions for mechanistic interpretability, where the idea is to explain black box ML models (like neural networks) with high-level algorithms that serve as causal hypotheses of the model’s behavior. The hypotheses are tested by learning an abstraction mapping between the ML model and the algorithm such that the properties of a causal abstraction hold. The main result of the paper is a theorem that proves that, without constraints on the abstraction map such as linearity, an abstraction map can be constructed between any ML model and any high-level hypothesis. The ensuing discussion emphasizes the implication that in the general case, this approach of explaining ML models can be vacuous. Experiments demonstrate that with nonlinear abstraction maps, interchange intervention accuracy remains consistently high.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Thm. 1 is an impactful result. Indeed, with growing literature in the space of causal abstractions for mechanistic interpretability, the non-linear representation dilemma really strikes at the heart of the issue of what assumptions are meaningful for determining causal hypotheses and what they imply in the behavior of neural networks.\n2. Assumptions are clearly stated and justified in Sec. 4.\n3. The paper is very well-written, with all notation clearly defined and well-labeled. The proof of Thm. 1 is carefully organized. The paper also neatly organizes all of the important background information, including the general process of using causal abstraction for mechanistic interpretability, the discussion of how information is encoded in deep neural networks, and some new definitions surrounding distributed abstractions that grounds prior work in the context of this paper.\n4. The experimental results show some interesting analyses. It is notable that in linear alignments, IIA decreases in deeper layers, a problem that goes away with nonlinear alignments. The fact that IIA increases with simpler alignments following more training is also interesting.\n\n**Weaknesses:**\n\n5. The paper could benefit from having figures that describe the causal abstraction for mechanistic interpretability procedure and highlight the impact of Thm. 1.\n6. The paper does not offer many new novel contributions and in some ways serves as a review or position paper. All content until Sec. 4 is background information (although Sec. 3.2 offers new definitions that portray existing work in new perspectives). The main contributions of the paper are Thm. 1 (with proof in the appendix) and the experimental results. The contributions are impactful, but they are mainly adding to discussion points surrounding the efficacy of this mechanistic interpretability approach rather than offering new solutions.\n\nOverall the paper is quite strong, and I recommend it for acceptance. I do not believe that the weaknesses I have discussed warrant a lower score.\n\nQUESTIONS:\n7. To what extent does Thm. 1 imply the problem is caused by the linearity assumption of the alignment rather than other factors? For example, perhaps IIA is an imperfect metric of measuring alignment since IIA does not consider semantic meaning of the algorithm nodes.\n\n8. How does the topic of alignment relate to disentangled causal representation learning? As in, is the idea of learning the alignment map similar to the idea of learning a representation map that disentangles the inputs into latent causal factors?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe most popular formalization of interpretability so far is the theory of _Causal abstractions_. They define a neural network (NN) as a complex computational graph, and an interpretation (G, m) as a simple (abstract) computational graph G, plus a map m of the graph's nodes and edges map to the NN's. Then, the interpretation (G, m) is a good one if there exists a mapping \\tau that maps values of the NN's nodes to their corresponding abstract graph nodes; and causally intervening in both NN and abstract graph A in the same way provides correct results.\n\nIt is well known that if \\tau is trivial, then causal abstraction is pretty useless. For example, if \\tau_n(x) = 0 for all nodes, all causal interventions produce the desired result and the graph is considered good.\n\nThe authors of this forcefully argue for the notion that, if we allow \\tau to be arbitrarily powerful, causal abstractions are also vacuous. The theoretical argument centers on \\tau partitioning NN activations into arbitrary groups (gerrymandering, if you will) that correspond to the algorithm's abstract values, which can be done regardless of whether the NN implements the algorithm A, so long as the NN has sufficiently many states. The practical argument centers on fitting a couple of well-known interpretability 'found' algorithms into completely unrelated or even randomly initialized networks, by using an up-to-8-layer NN as the map \\tau.\n\nThe authors then conclude that restricting \\tau (to be, for example, linear) works well.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nThe paper points out a critical problem with causal abstractions that many people that work with them do not think of, but that can pose problems. In practice basically all papers assume that the mapping \\tau is simple in some way, so it does not come up much. However, it's still an assumption that practitioners are making without being aware of, which could go away soon; for example if causal abstraction is used as a metric for automatic interpretability. For this reason, I think this is a fundamentally significant and original paper and all of its weaknesses are about presentation or about mistakes in the argument that make it not quite go through; but which can fixed easily.\n\nHere's my own experience reading the paper: I highlighted several chunks of the abstract, scribbling notes in disbelief, like \"Surely it's impossible to map [arbitrary] models to [arbitrary] algo[rithm]s\", implying that instead it's only possible to map arbitrary algorithms to NN models, because algorithms are simple whereas models are much complicated. I had also forgotten that causal abstraction only specifies that you can map high-level to low-level model, but not necessarily the opposite. If you were forced to map low-level to high-level model, then surely that map doesn't work for arbitrary low-level models? I think that _is_ true and made me believe this paper relatively quickly, but for the wrong reasons; I then thought that this paper spent too long on proofs for a relatively trivial statement. I read it again and realized the statement was not so trivial.\n\n# Weaknesses\n\n**My score is low because I want my comments to be addressed, but they should be easy to address; so I am hopeful that I will be able to recommend this paper to be accepted.** Here are the weaknesses, all fixable:\n\n## Section 4, Theorem 1's assumption: input-injective NNs\n\nI think the discussion in lines 234-237 of the \"DNNs are input-injective\" assumption is pretty misleading. First, this is clearly untrue: transformers use SiLUs or GeLUs which hit a negative peak at zero, and then go positive on both sides. ReLUs are also popular. Second: the citation of Morris et al. 2023 which is used to claim the assumption is \"well-approximated in practice\", only recovers 92% of 32-token inputs -- a far cry from 100%. Third, the authors themselves later write in lines 318-319 that this is not true in practice for their setup, and say this again in lines 407-410. Please be more upfront with this!\n\nAll of this needs to be said in Section 4 when discussing the assumptions. Input-injectivity is just not true. Which kind of invalidates the theorem. But in reality, NNs are ~close to input-injective, so they can be mapped to ~a large amount of algorithms that they don't implement. The theorem is still a good intuition builder.\n\n## Clarity: give intuition for Theorem 2\nIt would be nicer to give an intuition for the theorem. That is, if the NN is injective (it has a different state for every input sequence) and the algorithm is allowed to be arbitrarily complicated, then we can just gerrymander the hidden states to map to the abstract model. The hidden states need to be countable so there exists a computable algorithm that maps them to an abstract algorithm. (I haven't checked if this is the case thorougly, but the theorem does make use of partitions).\n\nQUESTIONS:\n# Section 4, Theorem 1's assumptions\n\n## Appendix F, proof for the injectivity assumption: Lemma 8\n\nBecause I was skeptical of transformers being input-injective, I dug into the proof, which line 237 claims exists in Appendix F.\n\nThe proof of this Theorem 2 uses Lemma 8 (\"MLPs are injective almost surely\") also does not assume linearities are injective.\n\nMy question is about the final line of the proof in section F.3. Each line seems to check out, except the last part in words in lines 1408-1411: \"Now, eq. 87 holds because the right-hand side is a constant while the left-hand side is a random variable drawn from a continuous distribution\". The left-hand-side is a row of the L layer's weights W_L, the right-hand-side is an expression involving the weights of other rows of W_L. **How is the RHS a constant?**\n\n## Assumption 5: The DNN solves the task T\n\nYou state in 247-248 that \"it would be impractical in practice to evaluate a NN that does not perform the task correctly. But that is exactly what you do in experiments! Fig 3, line 328-329 state that you're using random NNs.\n\n**Why does the theorem need this assumption?**\n\n## Assumption 3: output-surjectivity\nYou state that \"in practice, all outputs seem to still be producible by language models\" . This is directly contradicted the phenomenon of \"glitch tokens\" found by Rumbelow and Watkins (2023, https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation), where some token's unembedding has smaller norm than every other token in its direction, so there is no activation that will produce it as the maximum.\n\nAgain this shows that the theorem is false in practice, but it's *approximately* true in many circumstances and is a useful intuition pump. Please present it as such.\n\n# Figure 1: right plot, no perfect IIA?\nin the right-most plot in Figure 1, none of the alignment maps are able to get perfect IIA. Why do you think this is the case?\n\n# Arguments and remarks from page 9\n\nL379: claims that Burns et al. 2023 \"leverag[es] unsupervised probes\" in order to control the probe's complexity. In what way does unsupervised learning control the probe's complexity? Does that not come from the fact the probes are linear?\n\nL388-389, where the authors state they cannot formalize why Fig. 3 (right) inclines them to support the linear probe. That's OK, but what about an informal justification? One could say that it's provided in the previous sentence (\"\\phi^lin accompanies the actual DNN performance much more closely\"). If so, please just remove this sentence.\n\nLIMITATIONS:\nmostly. The authors addressed some of the limitationsI found in the final section. But I think they're problems with critical assumptions of the main theorem, so the assumptions should be addressed **when they're stated** and not at the end where readers might not find it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-22"}