# SLT for AI Safety

> This sequence draws from a [position paper](https://arxiv.org/abs/2502.05475) co-written with Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts, Susan Wei, Alexander Gietelink Oldenziel, Stan van Wingerden, George Wang, Zach Furman, Liam Carroll, Daniel Murfet. Thank you to Stan, Dan, and Simon for providing feedback on this post. 

**Alignment **$\subseteq$**Capabilities.** As of 2025, there is essentially no difference between the methods we use to align models and the methods we use to make models more capable. Everything is based on deep learning, and the main distinguishing factor is the choice of training data. So, the question is: what is the right data?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1f5de90b4c2ec7a0503dd77c46e07a31c173bb4d033c803f.png)

**Figure 1: Data differentiates alignment from capabilities.** Deep learning involves three basic inputs: (1) the architecture (+ loss function), (2) the optimizer, and (3) the training data. Of these, the training data is the main variable that distinguishes alignment from capabilities. 

**Alignment is data engineering.** Alignment training data specifies our values indirectly in the form of example demonstrations (instruction/safety fine-tuning), binary preferences (RLHF, DPO), or a constitution of abstract principles (constitutional AI, deliberative alignment). We train models against those underdetermined specifications and hope they internalize our intended constraints. 

**Alignment may require fundamental scientific progress.** As we've seen with recent examples of [sycophancy](https://openai.com/index/expanding-on-sycophancy/) and [reward-hacking](https://openai.com/index/chain-of-thought-monitoring/), our current indirect and empirical approach to alignment might not be enough to align increasingly powerful systems. These symptoms appear to stem from deeper scientific problems with deep learning:

1.  **The Problem of Generalization**: Distribution shifts can break learning-based constraints, so two models that behave similarly with respect to the training specification may generalize very differently in deployment. In particular, behaviors associated with general-purpose capabilities may generalize more robustly than alignment properties (the "[sharp left turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization)").
2.  **The Problem of Learning**: Even on a fixed distribution, the training process may learn an unintended solution. Our learning processes are noisy, heterogeneous, and not guaranteed to converge to an optimal solution for a given set of constraints. As a result, models may learn a dangerous simplification of our specifications, acquire unintended instrumental strategies (deception, incorrigibility, power-seeking, sandbagging, etc.), or suddenly change in ways that are hard to predict (and execute a "[treacherous turn](https://www.lesswrong.com/w/treacherous-turn)"). 

Past a certain level of capabilities, making further safety progress may require making scientific progress on some of these more fundamental questions. It is hard to align what you do not understand, and we still do not understand deep learning. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/315a94f7545a3fafa6b9e1a5f79278dbce33eb84d9267365.png)

**Figure 1: Key questions for a science of AI safety.** Many practical questions in AI safety are grounded in fundamental scientific questions about deep learning:   
1a. (Learning) How does training data determine the algorithms that models learn?  
1b. (Alignment) How can we choose training data to control what algorithms models learn?  
2a. (Generalization) How do learned algorithms generalize (under distribution shift)?  
2b. (Interpretability) How do a model internals enable (mis)generalization? 

**Singular learning theory for a science of alignment.** Singular Learning Theory (SLT) is a theory of Bayesian statistics that suggests the key to understanding deep learning is the geometry (specifically, the *degeneracies*) of the loss function and parameter-function map. SLT provides a starting point for understanding how learned algorithms underlie generalization and how training data determines those learned algorithms:

1.  **Interpretability:** The loss landscape around a given set of weights reflects the algorithms encoded by that set of weights and the sensitivity of those algorithms to changes in the data distribution \[[1](https://timaeus.co/research/2023-08-23-quantifying-degeneracy-copy), [4](https://timaeus.co/research/2025-04-25-susceptibilities), [6](https://timaeus.co/research/2025-04-10-programs-as-singularities)\]. If we can develop (a) the empirical tools to read this geometry and (b) the theoretical understanding to "invert" this reflection (a hypothesis we call "**Structural Bayesianism**"  \[[6](https://timaeus.co/research/2025-04-10-programs-as-singularities)\]), then we gain a principled framework for *interpreting* learned algorithms and linking this structure to generalization.
2.  **Alignment:** The algorithms a model learns are ultimately rooted in the model's training data: (a) Statistical structure in training data determines (b) the geometric structure in the loss landscape, which determines (c) the developmental structure in the learning process, which determines (d) the algorithmic structure in the final weights a model has learned. If we can (partially) "invert" this mapping (a hypothesis we call the "**S4 correspondence**", for four different kinds of "structure" \[[3](https://timaeus.co/research/2024-10-04-differentiation-and-specialization), [8](https://timaeus.co/research/2025-02-08-position)\]), then we can transform constraints on learned algorithms into interventions on training data. This provides a framework for *aligning* models through careful control of the training environment.

DL theory contains a number of important but mostly isolated ideas, such as the information bottleneck, NTK, saddle-to-saddle dynamics, links to statistical mechanics and QFT, mode connectivity, the study of SGD via high-dimensional probability, etc. Among these ideas, SLT seems one of the most important and one of the best bets for developing an eventual "unified theory" of deep learning, interpretability, and alignment. If successful, this would enable not just a better understanding of the theoretical problems of generalization and learning, but also the development of a new set of principled tools for "reading" and "writing" model internals. 

This progress would not be enough to guarantee safety on its own and would increase the risk burden on other areas in safety. However, fundamental progress may be necessary if we aren't on track to solve (inner) alignment by default.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/caadebba72346fdfaf1fe754d2d66ef2597400292db68d75.png)

**SLT for AI safety.** The loss landscape is where the training data, architecture, and optimizer interact. We expect that understanding the geometry of this landscape is equivalent to understanding internal structure, how it generalizes, and how to reliably control what structures arise. This enables a rigorous framework for advancing interpretability and alignment.

**Outline.** The rest of this sequence will survey recent progress in SLT \[[1](https://timaeus.co/research/2023-08-23-quantifying-degeneracy-copy), [2](https://timaeus.co/research/2024-02-04-develompental-landscape), [3](https://timaeus.co/research/2024-10-04-differentiation-and-specialization), [4](https://timaeus.co/research/2025-04-25-susceptibilities), [5](https://timaeus.co/research/2025-01-29-transient-structure), [6](https://timaeus.co/research/2025-04-10-programs-as-singularities), [7](https://timaeus.co/research/2025-04-25-modes), [8](https://timaeus.co/research/2025-02-08-position)\] and explore the potential applications for interpretability, alignment, and other areas of safety discussed above:

*   **SLT for Interpretability** (upcoming)
*   **SLT for Alignment** (upcoming)
*   **SLT for Present-Day Safety** (upcoming)

For more on the relevance of SLT to safety, see our [position paper](https://arxiv.org/abs/2502.05475) (co-written with a larger group of authors). For more on the technical fundamentals of SLT, see [Distilling Singular Learning Theory](https://www.lesswrong.com/s/czrXjvCLsqGepybHC) by Liam Carroll.