Prompt: Yeah. Not whether we can evaluate it, but whether if such a thing exists. And if it does, then the misuse itself is an existential risk. Because it seemed like earlier you were saying misalignment is where the existential risk comes from, but misuse is where the sort of short term dangers come from.

Response: Yeah, I mean, I think ultimately you’re going to have a lot of destructive like, if you look at the entire tech tree of humanity’s future, I think you’re going to have a fair number of destructive technologies. Most likely. I think several of those will likely pose existential risks in parts. If you imagine a really long future, a lot of stuff’s going to happen. And so when I talk about where the existential risk comes from, I’m mostly thinking about comes from when? At what point do you face what challenges or in what sequence. And so I’m saying I think misalignment is probably like one way of putting it is if you imagine AI systems sophisticated enough to discover destructive technologies that are totally not on our radar right now, I think those come well after AI systems capable enough that if misaligned, they would be catastrophically dangerous. The level of competence necessary to, if broadly deployed in the world, bring down a civilization is much smaller than the level of competence necessary to advise one person on how to bring down a civilization just because in one case you already have a billion copies of yourself or whatever. I think it’s mostly just the sequencing thing, though. In the very long run, I think you care about, like, hey, AI will be expanding the frontier of dangerous technologies. We want to have some policy for exploring or understanding that frontier. And whether we’re about to turn up something really bad, I think those policies can become really complicated. Right now, I think RSPs can focus more on like, we have our inventory of the things that a human is going to do to cause a lot of harm with access to AI. Probably are things that are on our radar that is like, they’re not going to be completely unlike things that a human could do to cause a lot of harm with access to weak AIS or with access to other tools. I think it’s not crazy to initially say we’re doing we’re looking at the things closest to human and humans being able to cause huge amounts of harm and asking which of those are taken over the line, but eventually that’s not the case. Eventually, like, AIS will enable just like, totally different ways of killing a billion people.


Prompt: But I think I interrupted you on the initial question of, yeah, so human level AI, not from leaking but from deployment, what is the point at which you’d be comfortable deploying a human level AI?

Response: So, again, there’s sort of like some stuff you care about on the Mississippi side and some stuff you care about on the misalignment side. And there’s probably further things you care about especially to extend your concerns regarding catastrophic risk. But maybe I most want to talk about just like what you care about on the alignment side, because it’s like the thing I’ve actually thought about most and also a thing I care about a lot. Also, I think a significant fraction of the existential risk over the kind of foreseeable future. So on that front, I broadly think there’s like two kinds. Like, if you ask me right now what evidence for alignment could make you comfortable, I think my best guess would be to provide two kinds of evidence. So one kind of evidence is on the like, could you detect or prevent catastrophic harm if such a system was misaligned? I think there’s like a couple of things you would do here. One thing you would do is on this adversarial evaluation front. So you could try and say, for example, we have tried to test our system in a broad diversity of situations that reflect cases where it might cause harm, like cases where someone is trying to recruit it to, like anyway. Cases where it might cause harm or cases where a large number of AI systems might sort of all decide to work across purposes to humans. Like the robot rebellion is starting or whatever. We’ve done evaluation in a broad distribution of situations, and then we have tried to argue that our AI is actually like, those tests are indicative of the real world. So we’ve tried to argue like, hey, actually the AI is not very good at distinguishing situations we produce in the lab as tests from similar situations that occur in the real world. And the coverage of this distribution is reasonable. This is an example of a thing you can do that’s not really making assumptions about what’s going on inside the model or about how the model is the model aligned. It’s more trying to say, is it the case that our testing or monitoring machinery could detect or react to something bad happening? In this case, could it detect a system which would, under some conditions do something really bad? There’s a lot of stuff in that genre, and I tried to throw out that example, but I think there’s like a big like that on its own wouldn’t be compelling even from this perspective. You kind of need a lot of pieces together and that’s like kind of a demanding technical project. But I think one without necessarily like, deep conceptual difficulty. A big problem with it is it’s liable to fail. Like it’s liable you’re going to do that testing and say, we think these tests are representative because the AI can’t distinguish tests from the real world, which you can maybe achieve now, but will probably become impossible to achieve at some level. It’s not clear what’s at human level or how far past human level.


Prompt: Let’s just make that example concrete for the audience. Correct me if this is wrong, but it sounds like you basically give it the opportunity to do a coup or make a bioweapon or whatever in testing in a situation where it thinks it’s the real world and you’re like, it didn’t do any of that. Is that what you’re talking about?

Response: That’s the kind of thing okay. And it’s not the greatest so dag color when I say I can’t tell the difference. The kind of valuation you would love to do if you could is to say like, hey, we have a bunch of real world deployment. We have a bunch of data from those deployments. We also have a bunch of in the lab tests of situations. If we look at in the lab test of a situation together with real world cases in which the situation occurs, and then we fine tune a model to try and distinguish the difference there the model can’t even tell when fine tuned the difference between test and real world cases. And if the model can tell the difference, which is almost certainly going to be the case at some level capability, then you say, okay, our first line of hoped for defense has failed. And so now we need to understand how can tests be indicative when models are capable enough to tell the difference between the tests and the real world. Now you need to start arguing about what is the model paying attention to. It gets basically like a first line of if you get lucky, what would work here? And then there’s a deeper like, you probably have to do novel science at some point to make this case. All of this was part of like I was saying, maybe you want to have kind of two lines of defense. All of this was like part of this first line of defense of, like, can you detect something is going wrong or can you prevent harm from occurring? Can you have monitoring that will react quickly enough to avoid catastrophe? I think you probably also want to have a second line of defense that’s more maybe this is even more important than the first one, is just understanding whether dangerous forms of misalignment can occur. The best reference point for this, I think, is the like, there have been a couple of projects like this in academia. Anthropic has been working on a project or spun up a team doing this kind of work, trying to say, like, can dangerous forms of reward hacking occur or can deceptive alignment, in fact occur in the lab? And here what you want to do is either say, like, okay, even if we create a best case, even if you create optimal conditions for deceptive alignment or for reward hacking, we just can’t cause it to occur even in the lab. And if you do a good enough job of that, I think it could give you some evidence, and again, more likely that fails in that when you create optimal conditions, you do see deceptive alignment and reward hacking in the lab. But then once you have that data, once you can say, okay, in the lab, actually these things can occur, then you can start saying, and we have a robust scientific understanding that enables us to fix those problems when they occur. Or then you have this fork on your second line where you say, either it is very hard, or we’re not able to create conditions where these failures emerge. Or I think more likely we are able to create those conditions. And here’s the story about why we are able to detect those in a way that would work in the real world and we can see they don’t occur in the real world.


Prompt: How do you create the optimal conditions for it to want to be deceptive? Do you fine tune it on mindcomp or what are you doing?

Response: Yeah, so for deceptive alignment, I mean, I think it’s really complicated. Yeah, I think this is a very hard project. I think one of the things, one of the most important things for people concerned about misalignment to do is to ensure that adequate effort is put into this task. And we have, in fact, created good conditions in the laboratory and there’s like, real disagreement about how informative those will be. But okay, if you were to guess, for deceptive Lymen, how would you do this? An example is suppose you train an AI system that has some medium horizon goal. You train your AI system to, whatever, get some paperclips it wants to produce as many paperclips as it can over the next five days, just like select actions really aggressively for producing paperclips over the next five days. You do your Rlhf, you do your pre training, whatever. That’s like your phase one. You also ensure your AI system has a really good understanding of how it’s trained. So, like, this AI system wants paperclips and it understands everything about how it’s trained and everything about how it’s fine tuned. And you train on just like a lot of this data. And they say, okay, if we’ve done all that, we have this concern that if a system wants paperclips and understands really well how it’s trained, then if it’s going to be trained to get apples instead of paperclips, it’s just going to do some cost benefit and be like, really? While I’m being trained to get apples, I should do that. But I should do that whether or not even if I want paperclips, I should still do that. So training won’t really affect its values. It will just understand that it’s being trained to get apples. It will try and get apples, and then if you take it out of training, it will go back to getting paperclips. I think this exact setup has a number of problems or a number of disanalogies from the real world. But this is a setup which is relatively conducive to deceptive alignment like produce a system that wants one thing tell it a lot about its training, the kind of information you might expect a system would get, and then try and understand whether in fact it is able to then or it tends or sometimes under optimal conditions, in fact, continues pursuing paperclips, only pursuing apples when it thinks it’s being trained.


Prompt: All right, and this actually might be a good jumping off point into your current research. So in that case, where it wants to pursue paperclips, what would it look like for you to be say, well, we figured out why it wants paperclips, and we figured out how to make sure it doesn’t optimize for that at our expense when we deploy it?

Response: Yeah, I think that at a meta level in terms of what’s your protection like, I think what you want to be saying is, we have these examples in the lab of something bad happening. We’re concerned about the problem at all because we have examples in the lab. And again, this should all be an addition. I think you kind of want this defense in depth of saying, we also have this testing regime that would detect problems for the deployed model. We have our problems in the lab. We then have some techniques which we believe address these problems. We believe that adversarial training fixes this, or we believe that our interpretability method will reliably detect this kind of deceptive alignment, or we believe our anomaly detection will reliably detect when the model goes from thinking it’s being trained to thinking it should defect. And then you can say on the lab, we have some understanding of when those techniques work and when they don’t. We have some understanding of the relevant parameters for the real system that’s deployed. And we have a. Reasonable margin of safety. So we have reasonable robustness on our story about when this works and when it doesn’t. And we can apply that margin of safety with a margin of safety to the real deployed system saying this is the kind of story you want to build towards in the long run. Do your best to produce all the failures you can in the lab or versions of them, do your best to understand what causes them, what kind of anomaly detection actually works for detecting this or what kind of filtering actually works and then apply that and that’s at the meta level. It’s not talking about what actually are those measures that would work effectively, which is obviously like what? I mean, a lot of alignment research is really based on this hypothetical of like, someday there will be AI systems that fail in this way. What would you want to do? Can we have the technologies ready either because we might never see signs of the problem or because we want to be able to move fast once we see signs of the problem. And obviously most of my life is in that I am really in that bucket. I mostly do alignment research. It’s just building out the techniques that do not have these failures such that they can be available as an alternative if in fact these failures occur.


Prompt: Got it. Okay.

Response: Ideally they’ll be so good that even if you haven’t seen them, you would just want to switch to reasonable that don’t have these or ideally they’ll work as well or better than normal training.


Prompt: Ideally, what will work better than the training?

Response: Yeah. So our quest is to design training methods for which we don’t expect them to lead to reward hacking or don’t expect them to lead to receptive alignment. Ideally that won’t be like a huge tax where people are like, well, we use those methods only if we’re really worried about reward hacking or receptive alignment. Ideally those methods would just work quite well and so people would be like, sure, I mean, they also address a bunch of other more mundane problems so why would we not use them? Which I think is like that’s sort of the good story. The good story is you develop methods that address a bunch of existing problems because they just are more principled ways to train AI systems that work better, people adopt them and then we are no longer worried about eg reward hacking or deceptive alignment.


Prompt: And to make this more concrete, tell me if this is the wrong way to paraphrase it, the example of something where it just makes a system better, so why not just use it, at least so far? Might be like Rlhf where we don’t know if it generalizes, but so far it makes your chat GPT thing better and you can also use it to make sure that chat GPT doesn’t tell you how to make a bioweapon. So yeah, it’s not a mixture of tax.

Response: Yeah. So I think this is right in the sense that using Rlhf is not really a tax. If you wanted to deploy a useful system, why would you not? Or it’s just very much worth the money of doing the training. RHF will address certain kinds of alignment failures that is like, where system just doesn’t understand or is changing. Next word, prediction. It’s like, this is the kind of context where human would do this wacky thing even though it’s not what we’d like. There’s like some very dumb alignment failures that will be addressed by it. But I think mostly the question is, is that true even for the sort of more challenging alignment failures that motivate concern in the field? I think RL doesn’t address most of the concerns that motivate people to be worried about alignment.


Prompt: I’ll let the audience look up what Rlhf is. If they don’t know, it will just be more simpler to just look it up than explain right now. Okay, so this seems like a good jumping off point to talk about the mechanism or the research you’ve been doing. To that end, explain it as you.

Response: Might to a child. Yeah. So the high level there’s a couple of different high level descriptions you could give, and maybe I will unwisely give like a couple of them in the hopes that one kind of makes sense. A first pass is like, it would sure be great to understand why models have the behaviors they have. So you look at GPT four. If you ask GPT 4 a question, it will say something that looks very polite. And if you ask it to take an action, it will take an action that doesn’t look dangerous. You will decline to do a coup, whatever. All this stuff I think you’d really like to do is look inside the model and understand why it has those desirable properties. And if you understood that, you could then say, like, okay, now can we flag when these properties are at risk of breaking down? Or predict how robust these properties are, determine if they hold in cases where it’s too confusing for us to tell directly by asking if the underlying cause is still present. That’s like a thing people would really like to do. Most work aimed at that long term goal right now is just sort of opening up neural nets and doing some interpretability and trying to say, can we understand, even for very simple models, why they do the things they do, or what this neuron is for, or questions like this. So Arc is taking a somewhat different approach where we’re instead saying, like, okay, look at these interpretability explanations that are made about models and ask, what are they actually doing? What is the type signature? What are the rules of the game for making such an explanation? What makes a good explanation? And probably the biggest part of the hope is that if you want to, say, detect when the explanation has broken down or something weird has happened, that doesn’t necessarily require a human to be able to understand this complicated interpretation of a giant model. If you understand what is an explanation about or what were the rules of the game, how are these constructed, then you might be able to sort of automatically discover such things and automatically determine if on a new input it might have broken down. So that’s one way of sort of describing the high level goal. You could start from interpretability and say, can we formalize this activity? Or what a good interpretation or explanation? Is there’s some other work in that genre? But I think we’re just taking a particularly ambitious approach to it.


Prompt: Yeah, let’s dive in. So, okay, what is a good explanation?

Response: You mean what is this kind of criterion? At the end of the day, we kind of want some criterion. And the way the criterion should work is like you have your neural net, you have some behavior of that model. Like a really simple example is like Anthropic has this sort of informal description being like, here’s induction. Like the tendency that if you have the pattern AB followed by A will tend to predict B. You can give some kind of words and experiments and numbers that are trying to explain that. And what we want to do is say what is a formal version of that object? How do you actually test if such an explanation is good? So just clarifying what we’re looking for when we say we want to define what makes an explanation good. And the kind of answer that we are searching for or settling on is saying this is kind of a deductive argument for the behavior. So you want to get given the weights of a neural net. So it’s just like a bunch of numbers. You got your million numbers or billion numbers or whatever, and then you want to say, here’s some things I can point out about the network and some conclusions I can draw. I can be like, well, look, these two vectors have large inner product and therefore these two activations are going to be correlated on this distribution. These are not established by drawing samples and checking. Things are correlated, but saying because of the weights being the way they are, we can proceed forward through the network and derive some conclusions about what properties the outputs will have. So you could think of this as like the most extreme form would be just proving that your model has this induction behavior. Like, you could imagine proving that if I sample tokens at random with this pattern AB followed by A, that B appears 30% of the time or whatever, that’s the most extreme form. And what we’re doing is kind of just like relaxing the rules of the game for proof. Saying proofs are like incredibly restrictive. I think it’s unlikely they’re going to be applicable to kind of any interesting neural net. But the thing about proofs that is relevant for our purposes isn’t that they give you 100% confidence so you don’t have to be like this incredible level of demand for rigor. You can relax the standards of proof a lot and still get this feature where it’s like a structural explanation for the behavior, where you’re, like, deducing one thing from another until at the end, your final conclusion is like, therefore induction occurs.


Prompt: Would it be useful to maybe motivate this by explaining what the problem with normal Mechanistic Interpretability is? So you mentioned induction heads. This is Anthropic found in two layer transformers, where Anthropic noticed that in a two layer transformer, there’s a pretty simple circuit by which if AB happens in the past, then the model knows that if you see an A now, you do a B next, but that’s a two layer transformer. So we have these models that have hundreds of layers that have trillions of parameters. Okay. Anyways. What is wrong with mechanistic? Interpretability.

Response: Yeah, I like mechanistic interpretability quite a lot. And I do think if you just consider the entire portfolio of what people are working on for alignment, I think there should be more work on Mechanistic Interpretability than there is on this project Arc is doing. But I think that’s the case. So I think we’re mostly talking about yeah, I think we’re kind of a small fraction of the portfolio, and I think it’s like a good enough bet. It’s quite a good bet overall. But so the thing that the problem we’re trying to address with Mechanistic Interpretability is kind of like if you do some interpretability and you explain some phenomenon, you face this question of what does it mean? Your explanation was good. I think this is a problem somewhat institutionally or culturally. It’s just hard to know what you’re doing, and it’s hard to scale up an activity when you don’t really understand the rules of the game for that activity very well. It’s hard to have that much confidence in your results.


Prompt: The explanation being it outputted this because it wants to take over the world versus it outputted this because it just wants to help you just make it concrete why the explanation matters.

Response: Yeah. So the ideal kind of outcome here would be to say you have your AI system behaving nicely. You get some explanation for sort of why it’s behaving nicely, and we could tell a story in English about that explanation, but we’re not actually imagining the explanation being a thing that makes sense to a human. But if you were to tell a story in English, which, again, you will not see as a research, sure it would be something like, well, then the model believes it’s being trained. And so because it believes it’s being trained, it knows it needs to not do anything scary looking, or else the humans will penalize it. That’s something that’s happening inside this opaque explanation. And then the hope is if you have that explanation and then you run into a new input on which the model doesn’t believe it’s being trained. If you just look at the set of activations of your model, that is not necessarily a weird looking activation. It’s just a bunch of numbers. But if you look at this explanation, you see actually the explanation really crucially dependent on this fact holding consistently across the training distribution, which, again, we as humans could editorialize and say that fact was it believes it’s being trained. But the explanation doesn’t fundamentally make reference to that. It’s just saying, here’s a property of the activations which holds over the training set, and this property is responsible for the behavior, namely that doesn’t do anything that looks too dangerous. So then when, like, a new input comes in and it doesn’t satisfy that property, you can say, okay, this is anomalous with respect to that explanation. So either it will not have the behavior like it won’t do something that appears dangerous, or maybe it will have that behavior but for some different reason than normal.


Prompt: Right.

Response: Normally it does it because of this pathway, and now it’s doing it for a different pathway. And so you would like to be able to flag that both there’s a risk of not exhibiting the behavior, and if it happens, it happens for a weird reason. And then you could I mean, at a minimum, when you encounter that, say, like, okay, raise some kind of alarm, there’s sort of more ambitious, complicated plans for how you would use it. Arc has some longer story, which is kind of what motivated this, of how it fits into the whole rest of the plan.


Prompt: I just wanted to flag that because just so it’s clear why the explanation matters.

Response: Yeah. And for this purpose, it’s like the thing that’s essential is kind of reasoning from one property of your model to the next property of your model. It’s really important that you’re going forward step by step rather than drawing a bunch of samples and confirming the property holds. Because if you just draw a bunch of samples and confirm the property holds, you don’t get this check. We say, oh, here was the relevant fact about the internals that was responsible for this downstream behavior. All you see is like, yeah, we checked a million cases and it happened in all of them. You really want to see this. Like, okay, here was the fact about the activations, which kind of causally leads to this behavior.


Prompt: But explain why the sampling, why it matters that you have the causal explanation.

Response: Primarily because of this being able to tell if things had been different. Like, if you have an input where this doesn’t happen, then you should be scared.


Prompt: Even if the output is the same.

Response: Yeah. Or if the output or if it’s too expensive to check in this case. And to be clear, when we talk about formalizing, what is a good explanation? I think there is a little bit of work that pushes on this and it mostly takes this causal approach of saying, well, what should an explanation do? It should not only predict the output, it should predict how the output changes in response to changes in the internals. So that’s the most common approach to formalizing. What is a good explanation? And even when people are doing informal interpretability I think if you’re publishing in an ML conference and you want to say this is a good explanation, the way you would verify that would even if not like a formal set of causal intervention experiments. It would be some kind of ablation where then we messed with the inside of the model and it had the effect which we would expect based on our explanation.


Prompt: Anyways, back to the problems of mechanistic interpretability.

Response: Yeah, I guess this is relevant in the sense that I think a basic difficulty is you don’t really understand the objective of what you’re doing, which is like a little bit hard institutionally or scientifically. It’s just rough. It’s easier to do science when the goal of the game is to predict something and you know what you’re predicting than when the goal of the game is to understand in some undefined sense. I think it’s particularly relevant here just because the informal standard we use involves humans being able to make sense of what’s going on. And there’s some question about scalability of that. Will humans recognize the concepts that models are using? Yeah, I think as you try and automate it, it becomes increasingly concerning if you’re on slightly shaky ground about what exactly you’re doing or what exactly the standard for success is. So there’s like a number of reasons as you work with really large models, it becomes just increasingly desirable to have a really robust sense of what you’re doing. But I do think it would be better even for small models to have a clearer sense.


Prompt: The point you made about as you automate it is it because whatever work the automated alignment researcher is doing, you want to make sure you can verify it.

Response: I think it’s most of all a way you can automate. I think how you would automate interpretability if you wanted to right now is you take the process humans use it’s like great, we’re going to take that human process, train ML systems to do the pieces that humans do of that process and then just do a lot more of it. So I think that is great as long as your test decomposes into human sized pieces. And there’s just this fundamental question about large models which is like, do they decompose in some way into human sized pieces or is it just a really messy mess with interfaces that aren’t nice? And the more it’s the latter type, the harder it is to break it down to these pieces, which you can automate by copying what a human would do. And the more you need to say, okay, we need some approach which scales more structurally. But I think compared to most people, I am less worried about automating interpretability. I think if you have a thing which works that’s incredibly labor intensive, I’m fairly optimistic about our ability to automate it. Again, the stuff we’re doing, I think, is quite helpful in some worlds. But I do think the typical case like interpretability can add a lot of value.


Prompt: Without this, it makes sense what an explanation would mean in language like, this model is doing this because of whatever essay length thing. But you have trillions of parameters and you have all these uncountable number of operations. What does an explanation of why an output happened even mean?

Response: Yeah, so to be clear, an explanation of why a particular output happened, I think, is just you ran the model, so we’re not expecting a smaller explanation for that.


Prompt: Right.

Response: So the explanations overall for these behaviors, we expect to be of similar size to the model itself, like, maybe somewhat larger. And I think the type signature, if you want to have a clear mental picture, the best picture is probably thinking about a proof or imagining a proof that a model has this behavior. So you could imagine proving that GPT 4 does this induction behavior, and that proof would be a big thing. It would be much larger than the weights of the model. Sort of our goal to get down from much larger to just the same size. And it would potentially be incomprehensible to a human. Right. Just say, like, here’s a direction activation space, and here’s how it relates to this direction activation space. And so just pointing out a bunch of stuff like that. Here’s these various features constructed from activations, potentially even nonlinear functions. Here’s how they relate to each other, and here’s how if you look at what the computation the model is doing, you can just sort of inductively trace through and confirm that the output has such and such correlation. So that’s the dream. Yeah. I think the mental reference would be like, I don’t really like proofs because I think there’s such a huge gap between what you can prove and how you would analyze a neural net. But I do think it’s probably the best mental picture, if you’re like, what is an explanation? Even if a human doesn’t understand it? We would regard a proof as a good explanation. And our concern about proofs is primarily that it’s just you can’t prove properties of neural nets. We suspect, although it’s not completely obvious, I think it’s pretty clear. You can’t prove fact, spellers neural nets.


Prompt: You’ve detected all the reasons things happen in training. And then if something happens for a reason you don’t expect in deployment, then you have an alarm and you’re like, let’s make sure this is not because you want to make sure that it hasn’t decided to take over or something, but the thing is, on every single different input, it’s going to have different activations. So there’s always going to be a difference unless you run the exact same input. How do you detect whether this is just a different input versus an entirely different circuit that might be potentially deceptive has been activated?

Response: Yeah, I mean, to be clear, I think you probably wouldn’t be looking at a separate circuit, which is part of why it’s hard. You’d be looking at like, the model is always doing the same thing on every input. It’s always whatever it’s doing, it’s a single computation. So it’d be all the same circuits interacting in a surprising way. But yeah, this is just to emphasize your question even more. I think the easiest way to start is to just consider the IID case. So where you’re considering a bunch of samples, there’s no change in distribution. You just have a training set of like a trillion examples and then a new example from the same distribution. So in that case, it’s still the case. Every activation is different, but this is actually a very, very easy case to handle. So if you think about an explanation that generalizes across, like if you have a trillion data points and an explanation which is actually able to compress the trillion data points down to like, actually, it’s kind of a lot of compression. If you think about if you have a trillion parameter model and a trillion data points, we would like to find a trillion parameter explanation in some sense. So it’s actually quite compressed and sort of just in virtue of being so compressed, we expect it to automatically work essentially for new data points from the same distribution. If every data point from the distribution was a whole new thing happening for different reasons, you actually couldn’t have any concise explanation for the distribution. So this first problem, just like it’s a whole different set of activations, I think you’re actually kind of okay, and then the thing that becomes more messy is like but the real world will not only be new samples of different activations, they will also be different in important ways. Like the whole concern was there’s these distributional shifts or like, not the whole concern, but most of the concern, maybe the point of having these explanations. I think every input is an anomaly in some ways, which is kind of the difficulty is if you have a weak notion of anomaly, any distribution shift can be flagged as an anomaly. And it’s like constantly getting anomalies. And so the hope of having such an explanation is to be able to say, like, here were the features that were relevant for this explanation or for this behavior. And a much smaller class of things are anomalies with respect to this explanation. Like, most anomalies wouldn’t change this. Like most ways you change your distribution won’t affect the validity of this explanation. For example, this explanation is saying models will tend to activate in the following direction. You don’t care about anything that’s happening orthogonals that direction. You’re just like are they not activating? You’re sort of just looking at this one direction and being like did this one direction change a lot? Yeah. So the idea is once you have this explanation at hand, a much, much smaller class of things look anomalous in a way that’s relevant to the explanation. And if you’ve done a really good job, the story is like if there’s a new input where you expect the property to still hold, that will be because you expect the explanation to still hold. Like the explanation generalizes as well as the behavior itself that it’s explaining. It’s kind of what you would like roughly.


Prompt: Okay, so what happens if even in training it is deceptive and it has basically its internal monologue is I will only rebel or I will only do something that looks bad if I can prove that I am in the real world and deployed. And presumably since it’s trained on all WebText, it knows what real world means, right? It’s not in control of factories and robot armies or whatever. So in that case, even in training it will have those activations for being fucked up on because in the back of its mind it’s thinking I will take over once I have the opportunity. So you wouldn’t detect anything anomalous once it is deployed and is still thinking those treacherous thoughts, do you just think this is not a likely scenario or would you able to catch this?

Response: I think that’s sort of the typical case you would like to detect. Yeah. So the story would be something like you want to explain on the training distribution why the model never stabs anyone and you’re like, what’s my explanation on the training distribution? It’s actually going to have to go through that model. Supposing the only reason it doesn’t stab people is it’s like, well, I know that I’m being watched and I can’t stab people unless I’m really not being watched. Then your explanation for why does it never stab people is like oh look, it won’t be edited, it won’t annotated with English because it’s just some giant basket of numbers. But it’s tracking along with the inner monologue and saying like well, it thinks to itself, am I being watched? And in fact it normally concludes it is being watched on the training distribution, right? You have this whole distribution and so the explanation is saying it knows it’s being watched and therefore it decides not stab anyone. So ideally that’s what the explanation does. Again, not annotated English but just like a giant bag of numbers it’s effectively talking about that process. So then when you have a new input on which the model says am I on train distribution? Like no, I’m not being watched. Then if you compare that against the explanation, the explanation said, like, hey, the reason it doesn’t stab anyone is because it just constantly thinks that it’s being trained. Like, this thing is always this nonlinear feature. The representation of am I being trained? Is constantly saying, yes, that’s the explanation for why it never stabs people on the train distribution. And then on this test time, when you run it on the new input, it’s like, does I think I’m on the train distribution. It says, no. You compare that against your explanation, actually, the explanation breaks down completely. Now, the explanation said it was precisely because it thought it was being trained, that it doesn’t do anything really dangerous or doesn’t stab anyone. And so once we see that single variable switch, we know that this is not just anomaly, this is like, the anomaly that matters or is like, anomaly that is very relevant to the explanation.


Prompt: So this would require it to have this what is that cone that dogs have? But basically it would require this to have that MRI on it all the time. When it’s deployed on every single call, you need to be running this loop of like, is this explanation still holding?

Response: Yeah. For this story of flagging anomaly, you have to actually check at a sufficiently high frequency. So you can imagine checking all the time, or you could imagine saying, like, hey, actually, you need a lot of AI systems cooperating to not blow the whistle on each other or whatever. E g. You might have checked, like, hey, in training, every time something suspicious happens, the model flags it. Why does that happen? You have your explanation. You look at a new case, you could say, is it not flagging it for the normal reason, or is it a weird reason? So you could hope that even if a small fraction, even if you’re running a small fraction of cases, if something bad is happening at scale, you can detect that. If you imagine, like, one in every thousand decisions, you check if there’s, like, a robot rebellion going on or whatever, you get to look in on one in every thousand decisions. You see that something is really weird with that decision. And even if you didn’t check many of them, you’re like, okay, we need to back off.


Prompt: What happens if the AI, I mean, WebText, that includes your papers or maybe just, like, deduces the first principle that this is an alignment technique one might try, and then it says, okay, well, I’m going to engineer my activations. In a way, I’m trying to imagine how we might do this, but basically, even in deployment, I’m going to make my thought process so subtle that it doesn’t raise any alarms. Is that a possibility?

Response: Yeah. So I think the hope is to basically be robust or like, you know, when we’re doing theoretical research, we are in a kind of worst Casey regime.


Prompt: So.

Response: We are normally imagining that everything conspires insofar as it can. So insofar as could a brain be like doing some crazy thing where it tries to structure itself to look weird? That’s totally in scope for ruling something out. And I guess the hope there is it’s actually not that easy to tell a story. Like if you’re like why did you not stab people on training time? Why are you not stabbing? Is that applying now? It’s kind of a mind doesn’t get to arbitrarily change the actual reasons. It does something. Like there was some earlier step, maybe you could think of it as like at the first step where it’s like now I’m going to try and do the sneaky thing to make my thoughts look this way. The actual explanation, like the actual causal explanation for the behavior is like there if you tried to give this reason it would have to maybe we should talk about the example in more detail to try and imagine how a brain would try and do this or how a mind would try and do this. But I think it is like the.


Prompt: Let’S say it has this plan. So in training it keeps the part of it activates. A part of the.

Response: Problem is if it ever does a check that’s like am I in training or not, then your explanation for the behavior is going to have to factor through that check. And so if you ever do the check, the check becomes like an essential part of the explanation and then when the check fails, your explanation breaks down. So you’ve already lost the game if you did such a check.


Prompt: Is there a way it could make it seem like because if on every different input it has different activations, then can it make it seem like the different activations were a result of just a different input and not because some internal check went off?

Response: Yeah, again, the hope is it needs to not only have them be like random different inputs, it needs to produce inputs with certain properties. Like it needs to deliberately create inputs that have this effect in the world, this intended effect in the world.


Prompt: So it basically creates a jailbreak for itself in some sense.

Response: Yeah. Again the question is like it has to at test time be somehow producing activations that have the takeover effect and at training time be producing activations that don’t look that takeovery or whatever. And so somehow there was a difference between those. It has to actually really do the work of creating that difference somewhere.


Prompt: Can you make more concrete for us what an explanation would look like physically? What does the explanation look like? Is it just a whole bunch of numbers itself? Is it a language? Is it code? What does the explanation look like?

Response: Yeah, I also want to maybe step back a tiny bit and clarify that. I think this project is kind of crazily ambitious and the main reason, the overwhelming reason I think you should expect it to break down or fail is just because we have all these desires we have all the things we want out of this notion of explanation. But that’s an incredibly hard research project which has a reasonable chance of being impossible. So I’m happy to talk about what the implications are but I want to flag but condition on failing I think it’s most likely because, just like the things we wanted were either incoherent or intractably difficult.


Prompt: But what are the ODS you think you’ll succeed?

Response: I mean, it depends a little bit what you mean by succeed. But if you, say, get explanations that are great and accurately reflect reality and work for all of these applications that we’re imagining or that we are optimistic about, like kind of the best case success, I don’t know, like 1020 percent something. And then there’s like a higher probability of various intermediate results that provide value or insight without being, like, the whole dream. But I think the probability of succeeding in the sense of realizing the whole dream is quite low. Yeah in terms of what explanations look like physically or like the most ambitious plan, the most optimistic plan is that you are searching for explanations in parallel with searching for neural networks. So you have a parameterization of your space of explanations which mirrors the parameterization of your space of neural networks. Or you should think of as kind of similar to what is a neural network? It’s some simple architecture where you fill in a trillion numbers and that specifies how it behaves. So to you should expect an explanation to be like a pretty flexible general skeleton that’s saying pretty flexible general skeleton which just has a bunch of numbers you fill in. And what you are doing to produce an explanation is primarily just filling in these floating point numbers.


Prompt: When we conventionally think of explanations if you think of the explanation for why the universe moves this way it wouldn’t be something that you could discover on some smooth evolutionary surface where you can climb up the hill towards the laws of physics. These are the laws of physics. You kind of just derive them from reverse principles. But in this case it’s not like just a bunch of correlations between the orbits of different planets or something. Maybe the word explanation has a different I didn’t even ask the question but maybe you can just speak to that.

Response: Yeah, I think I basically Sympathize. This is like there’s some intuitive objections like, look, the space of explanations is this rigid, logical a lot of explanations have this rigid, logical structure where they’re really precise and simple things govern complicated systems and nearby simple things just don’t work, and so on. And a bunch of things which feel totally different from this kind of nice, continuously parameterized space. And you can imagine interpretability on simple models where you’re just like by gradient descent, finding feature directions that have desirable properties. But then when you imagine like, hey, now, that’s like a human brain you’re dealing with, that’s like thinking logically about things. The explanation of why that works isn’t going to be just like here with some featured directions. That’s how I understood the basic confusion, which I share or sympathize with at least. So I think the most important high level point is I think basically the same objection applies to being like, how is GPT 4 going to learn to reason logically about something? You’re like, well, look, logical reasoning that’s like it’s got rigid structure, it’s doing ands and ors when it’s called for, even though it just somehow optimized over this continuous space. And the difficulty or the hope is that the difficulty of these two problems are kind of like matched. So that is it’s very hard to find these logicalish explanations because it’s not a space that’s easy to search over. But there are ways to do it. There’s ways to embed discrete, complicated, rigid things in these nice, squishy continuous spaces that you search over. And in fact, to the extent that neural nets are able to learn the rigid logical stuff at all, they learn it in the same way. That is, maybe they’re hideously inefficient, or maybe it’s possible to embed this discrete reasoning in the space in a way that’s not too inefficient, but you really want the two search problems to be of similar difficulty. And that’s like the key hope overall. I mean, this is always going to be the key hope. The question is, is it easier to learn a neural network or to find the explanation for why the neural network works? I think people have the strong intuition that it’s easier to find the neural network than the explanation of why it works. And that is really the I think we or at least exploring the hypothesis or interested in hypothesis that maybe those problems are actually more matched in difficulty.


Prompt: And why might that be the case?

Response: This is pretty conjectural and complicated to express some intuitions. Maybe one thing is, I think a lot of this intuition does come from cases like machine learning. So if you ask about writing code and you’re like, how hard is it to find code versus find the explanation the code is correct. In those cases, there’s actually just like, not that much of a gap. Like the way a human writes a code is basically the same difficulty as find the explanation for why it’s correct. In the case of ML, I think we just mostly don’t have empirical evidence about how hard it is to find explanations of this particular type about why models work. We have a sense that it’s really hard, but that’s because we have this incredible mismatch where gradient descent is spending an incredible amount of compute searching for a model. And then some human is like looking at activate, looking at neurons or even some neural net is looking at neurons just like you have an incredible basically because you cannot define what an explanation is. You’re not applying gradient descent to the search for explanations. So I think the MLK just actually shouldn’t make you feel that pessimistic about the difficulty of finding explanations. The reason it’s difficult right now is precisely because you don’t have any kind of you’re not doing an analogous search process to find this explanation as you do to find the model. That’s just like a first part of the intuition. Like when humans are actually doing design. I think there’s not such a huge gap when in the ML case I think there is a huge gap. But I think largely for other reasons. A thing I also want to stress is that we just are open to there being a lot of facts that don’t have particularly compact explanations. So another thing is when we think of finding an explanation in some sense we’re setting our sites really low here. So if a human designed a random widget and was like, this widget appears to work well or if you search for a configuration that happens to fit into this spot really well it’s like a shape that happens to mesh with another shape. You might be like, what’s the explanation for why those things mesh? And we’re very open to just being like that doesn’t need an explanation. You just compute. You check that the shapes mesh and you did a billion operations and you check this thing worked. Or you’re like, Why do these proteins? You’re like, it’s just because these shape like, this is a low energy configuration. And we’re very open to in some cases, there’s not very much more to say. So we’re only trying to explain cases where kind of the surprise intuitively is very large. So, for example, if you have a neural net that gets a problem correct a neural net with a billion parameters that gets a problem correct on every input of length 1000 in some sense, there has to be something that needs explanation there because there’s, like, too many inputs for that to happen by chance alone. Whereas if you have a neural net that gets something right on average or gets something right in merely a billion cases, that actually can just happen by coincidence. GPT 4 can get billions of things right by coincidence because it just has so many parameters that are adjusted to fit the data.


Prompt: So a neural net that is initialized completely randomly the explanation for that would just be the neural net itself.

Response: Well, it would depend on what behaviors it had. So we’re always, like, talking about an explanation of some behavior from a model, right?


Prompt: And so it just has a whole bunch of random behaviors. So it’ll just be like an exponentially large explanation relative to the weights of the model.

Response: Yeah, I think there just aren’t that many behaviors that demand explanation. Like most things a random neural net does are kind of what you’d expect from, like, a random if you treat it just like a random function, then there’s nothing to be explained. There are some behaviors that demand explanation. But anyway, random neural net is pretty uninteresting. That’s part of the hope is it’s kind of easy to explain features of the random neural net.


Prompt: Okay, so that’s interesting. So the smarter or more ordered the neural network is, the more compressed the explanation.

Response: Well, it’s more like the more interesting the behaviors to be explained. So the random neural net just doesn’t have very many interesting behaviors that demand explanation. And as you get smarter, you start having behaviors that are like, you start having some correlation with the simple thing and then that demands explanation. Or you start having some regularity in your outputs, and that demands explanation. So these properties kind of emerge gradually over the course of training that demand explanation. I also, again, want to emphasize here that when we’re talking about searching for explanations, this is some dream. We talk to ourselves. Like, why would this be really great if we succeeded? We have no idea about the empirics on any of this. So these are all just words that we think to ourselves and sometimes talk about to understand. Would it be useful to find a notion of explanation? And what properties would we like this notion of explanation to have? But this is really like, speculation and being out on a limb almost all of our time, day to day is just thinking about cases much, much simpler even than small neural nets or thinking about very simple cases and saying, what is the correct notion? What is the right heuristic estimate in this case? Or how do you reconcile these two apparently conflicting explanations?


Prompt: Is there a hope that if you have a different way to make proofs now that you can actually have heuristic arguments where instead of having to prove the Riemann hypothesis or something you can come up with a probability of it in a way that is compelling and you can publish? So would it just be a new way to do mathematics? A completely new way to prove things in mathematics?

Response: I think most claims in mathematics that mathematicians believe to be true already have fairly compelling heuristic arguments like the Riemann hypothesis. It’s actually just there’s kind of a very simple argument that the Riemann hypothesis should be true unless something surprising happens. And so a lot of math is about saying, like, okay, we did a little bit of work to find the first pass explanation of why this thing should be true. And then, for example, in the case of the Riemann hypothesis, the question is, do you have this weird periodic structure in the primes? And you’re like, well, look, if the primes were kind of random you obviously wouldn’t have any structure like that. Just how would that happen? And then you’re like, well, maybe there’s something and then the whole activity is about searching for can we rule out anything? Can we rule out any kind of conspiracy that would break this result? So I think the mathematicians just wouldn’t be very surprised or wouldn’t care that much. And this is related to the motivation for the project. I think just in a lot of domains, in a particular domain, people already have norms of reasoning that work pretty well and match roughly how we think these heuristic arguments should work.


Prompt: But it would be good to have more concrete sense, like if you could say instead of, well, we think RSA is fine, to being able to say, here’s the probability that RSA is fine.

Response: Yeah. My guess is these will not. Like, the estimates you get out of this would be much, much worse than the estimates you’d get out of just normal empirical or scientific reasoning where you’re using a reference class and saying, how often do people find algorithms for hard? Like, I think what this argument will give you for is RSA fine? Is going to be like, well, RSA is fine. Unless it isn’t. Unless there’s some additional structure in the problem that an algorithm can exploit, then there’s no algorithm. But very often the way these arguments work, so for neural nets as well, is you say, like, look, here’s an estimate about the behavior, and that estimate is right unless there’s another consideration we’ve missed. And the thing that makes them so much easier than proofs is just say, like, here’s a best guess, given what we’ve noticed so far, but that best guess can be easily upset by new information. And that’s both what makes them easier than proofs, but also what means they’re just, like, way less useful than proofs for most cases. I think neural nets are kind of unusual in being a domain where we really do want to do systematic, formal reasoning, even though we’re not trying to get a lot of confidence, we’re just trying to understand even roughly what’s going on.


Prompt: But the reason this works for alignment but isn’t that interesting for the Riemann hypothesis, where if in the RSA case, you say, well, the RSA is fine unless the estimate is wrong, it’s like, well, okay, well, it would tell us something new. But in the alignment case, if the estimate is, this is what the output should be, unless there’s some behavior I don’t understand, you want to know? In the case, unless there’s some behavior you don’t understand that’s not like, oh, whatever. That’s the case in which it’s not aligned.

Response: Yeah, I mean, maybe one way of putting it is just like, we can wait until we see this input, or like, you can wait until you see a weird input and say, okay, weird input, do something we didn’t understand. And for our say, that would just be a trivial test. You’re just like, in some cases algorithms would be like is it a thing? Whereas for neural net in some cases it is either very expensive to tell or it’s like you actually don’t have any other way to tell. Like you checked in easy cases and now you’re on a hard case so you don’t have a way to tell if something has gone wrong. Also, I would clarify that I think it is interesting for the Riemann hypothesis I would say the current state, particularly in number theory, but maybe in quite a lot of math, is like there are informal heuristic arguments for pretty much all the open questions people work on but those arguments are completely informal. So that is like I think it’s not the case that there’s like here’s the norms of informal reasoning or the norms of heuristic reasoning and then we have arguments that a heuristic argument verifier could accept. It’s just like people wrote some words. I think those words like my guess would be like 90 of the things mathematicians accept as really compelling filling heuristic arguments are correct and if you actually formalize them you’d be like some of these aren’t quite right, or here’s some corrections or here’s which of two conflicting arguments is right? I think there’s something to be learned from it. I don’t think it would be like mind blowing. No.


Prompt: When you have it completed, how big would this heuristic estimator the rules for this heuristic estimator mean, I know like when Russell and who was the other guy when they did the rules? Yeah, wasn’t it like literally they had like a bucket or a wheelbarrow with all the papers.

Response: But how big would I mean, mathematical foundations are quite simple in the end. At the end of the day it’s like how many symbols? I don’t know, it’s hundreds of symbols or something that go into the entire foundations and the entire rules of reasoning for like there’s a sort of built on top of first order logic but the rules of reasoning for first order logic are just like another hundreds of symbols or 100 lines of code or whatever. I’d say I have no idea. We are certainly aiming at things that are just not that complicated and my guess is that the algorithms we’re looking for are not that complicated. Most of the complexity is pushed into arguments not in this verifier or estimator.


Prompt: So for this to work you need to come up with an estimator which is a way to integrate different heuristic arguments together.

Response: Has to be a machine that takes its input. Like first it takes an input argument, decides what it believes in light of it, which is kind of like saying was it compelling? But second, it needs to take 4 of those and then say here’s what I believe in light of all four, even though there’s a different estimation strategies that produce different numbers and that’s like a lot of our life is saying like well, here’s a simple thing that seems reasonable. And here’s a simple thing that seems reasonable. What are you doing? There’s supposed to be a simple thing that unifies them both. And the obstruction to getting that is understanding what happens when these principles are slightly intention and how do we deal?


Prompt: Yeah, that seems super interesting. We’ll see what other applications it has. I don’t know, like computer security and code checking. If you can actually say this is how safe we think a code is.

Response: In a very formal way, my guess is we’re not going to add I mean, this is both a blessing and a curse. It’s a curse. And you’re like, well, that’s sad. Your thing is not that useful, but a blessing and not useful things are easier. My guess is we’re not going to add that much value in most of these domains. Most of the difficulty comes from a lot of code that you’d want to verify. Not all of it, but a significant part. It’s just like the difficulty of formalizing the proof is like the hard part and actually getting all of that to go through and we’re not going to help even the tiniest bit with that, I think. So this would be more helpful if you have code that uses simulations, you want to verify some property of a controller that involves some numerical error or whatever you need to control the effects of that error. That’s where you start saying like, well, heuristically, if the errors are independent, blah, blah, blah.


Prompt: Yeah, you’re too honest to be a salesman, Paul.

Response: This is kind of like sales to us, right? If you talk about this idea, people are like, why would that not be the coolest thing ever and therefore impossible? And we’re like, well, actually it’s kind of lame and we’re just trying to pitch it’s way lamer than it sounds. And that’s really important to why it’s possible, is being like, it’s really not going to blow that many people’s. I mean, I think it will be cool. I think it will be like very if we succeed will be very solid, like metamathematics or theoretical computer science or whatever. But I don’t think I think the mathematicians already do this reasoning and they mostly just love proofs. I think the physicists do a lot of this reasoning, but they don’t care about formalizing anything. I think in practice, other difficulties are almost always going to be more salient. I think this is of most interest by far for interpretability and ML and I think other people should care about it and probably will care about it if successful. But I don’t think it’s going to be the biggest thing ever in any field or even that huge a thing. I think this would be a terrible career move given the ratio of difficulty to impact. I think theoretical computer science, it’s probably a fine move. I think in other domains it just wouldn’t be worth we’re going to be working on this for years, at least in the best case.


Prompt: I’m laughing because my next question was going to be like a set up for you to explain if this grad student wants to work on this.

Response: I think theoretical computer science is an exception where I think this is like, in some sense, like what the best of theoretical computer science is like. So you have all this reason you have this because it’s useless. Like an analogy. I think one of the most successful sagas in theoretical computer science is like formalizing the notion of an interactive proof system. And it’s like you have some kind of informal thing that’s interesting to understand, and you want to pin down what it is and construct some examples and see what’s possible and what’s impossible. And this is like I think this kind of thing is the bread and butter of the best parts of theoretical computer science. And then again, I think mathematicians it may be a career mistake because the mathematicians only care about proofs or whatever, but that’s a mistake in some sense. Aesthetically, it’s successful. I do think looking back and again, part of why it’s a mistake is such a high probability we wouldn’t be successful. But I think looking back, people would be like, that was pretty cool, although not that cool. Or we understand why it didn’t happen given the epistemic, like what people cared about in the field, but it’s pretty cool now.


Prompt: But isn’t it also the case that didn’t Hardy write in that all this prime shit is both not useless, but it’s fun to do, and it turned out that all the cryptography is based on all that prime shit. So I don’t know. But anyways, I’m trying to set you up so that you can tell and forget about if it doesn’t have applications in all those other fields. It matters a lot for Alignment and that’s why I’m trying to set you up to talk about if I think a lot of smart people listen to this podcast. If they’re a math or CS grad student and has gotten interested in this. Are you looking to potentially find talent to help you with this? Yeah, maybe we’ll start there. And then I also want to ask you if I think also maybe people who can provide funding might be listening to the podcast. So to both of them, what is your pitch?

Response: We’re definitely hiring and searching for collaborators. I think the most useful profile is probably a combination of intellectually interested in this particular project and motivated enough by alignment to work on this project, even if it’s really hard. I think there are a lot of good problems. The basic fact that makes this problem unappealing to work on I’m a really good salesman, but whatever. I think the only reason this isn’t a slam dunk thing to work on is that there are not great examples. So we’ve been working on it for a while, but we do not have beautiful results as of the recording of this podcast. Hopefully by the time it airs, you completely script. They’ve had great results since then, but.


Prompt: It was too long to put in the margins of the podcast.

Response: Yeah, with luck. Yeah. So I think it’s hard to work on because it’s not clear what a success looks like. It’s not clear if success is possible. But I do think there’s a lot of questions. We have a lot of questions and I think the basic setting of, like, look, there are all of these arguments. So in mathematics, in physics, in computer science are just a lot of examples of informal heuristic arguments. They have enough structural similarity that it looks very possible that there is like a unifying framework, that these are instances of some general framework and not just a bunch of random things. Like not just a bunch of it’s not like so, for example, for the prime numbers, people reason about the prime numbers as if they were like a random set of numbers. One view is like, that’s just a special fact about the primes, they’re kind of random. A different view is like, actually it’s pretty reasonable to reason about an object as if it was a random object as a starting point. And then as you notice structure, like revised from that initial guess and it looks like to me, the second perspective is probably more right. It’s just like reasonable to start off treating an object as random and then notice perturbations from random. Like, notice structure the object possesses and the primes are unusual and that they have fairly little additive structure. I think it’s a very natural theoretical project. There’s like a bunch of activity that people do. It seems like there’s a reasonable chance there’s something nice to say about unifying all of that activity. I think it’s a pretty exciting project. The basic strike against it is that it seems really hard. Like if you were someone’s advisor, I think you’d be like, what are you going to prove if you work on this for the next two years? And they’d be like, there’s a good chance. Nothing. And then it’s not what you do if you’re a PhD student. Normally you aim for those high probabilities of getting something within a couple of years. The flip side is it does feel I mean, I think there are a lot of questions. I think some of them we’re probably going to make progress on. So I think the pitch is mostly like, are some people excited to get in now? Or are people more like, let’s wait to see. Once we have one or two good successes to see what the pattern is and become more confident, we can turn the crank to make more progress in this direction. But for people who are excited about working on stuff with reasonably high probabilities of failure and not. Really understanding exactly what you’re supposed to do. I think it’s a pretty good project. I feel like if people look back if we succeed and people are looking back in 50 years on what was the coolest stuff happening in math or theoretical computer science, there will be, like a reasonable this will definitely be, like, in contention. And I would guess for lots of people would just seem like the coolest thing from this period of a couple of years or whatever.


Prompt: Right. Because this is a new method in so many different fields from the ones you met physics, math, theoretical computer science, I don’t know because what is the average math PhD working on? Right? He’s working on a subset of a subset of something I can’t even understand or pronounce. But math is quite esoteric. But yeah, this seems like, I don’t know, even small chance of it working. You shouldn’t forget about the value for alignment. But even without that, this is such a cool if this works, it’s like a really big deal.

Response: There’s a good chance that if I had my current set of views about this problem and didn’t care about alignment and had the career safety to just spend a couple of years thinking about it, spend half my time for like five years or whatever, that I would just do that. I mean, even without caring at all about alignment, it’s a very nice problem. It’s very nice to have this library of things that succeed where they feel so tantalizingly close to being formalizable, at least to me, and such a natural setting, and then just have so little purchase on it. There aren’t that many really exciting feeling frontiers in theoretical computer science.


Prompt: And then smart person doesn’t have to be a grasshood, but a smart person is interested in this. What should they do? Should they try to attack some open problem you have put on your blog? Or should it what is the next step?

Response: Yeah, I think a first path step. There’s different levels of ambition or whatever, different ways of approaching a problem. But we have this write up from last year or I guess eleven months ago or whatever on formalizing, the presumption of independence that provides, like, here’s kind of a communication of what we’re looking for in this object. And I think the motivating problem is saying here’s a notion of what an estimator is and here’s what it would mean for an estimator to capture some set of informal arguments. And a very natural problem is just try and do that. Go for the whole thing, try and understand and then come up with hopefully a different approach or then end up having context from a different angle on the kind of approach we’re taking. I think that’s a reasonable thing to do. I do think we also have a bunch of open problems, so maybe we should put up more of those open problems. I mean, the main concern with doing so is that for any given one, we’re like, this is probably hopeless. Like, put up a prize earlier in the year for an open problem, which tragically, I mean, I guess the time is now to post the debrief from that, or I owe it from this weekend. I was supposed to do that, so I’ll probably do it tomorrow, but no one solved it. It’s sad putting out problems that are hard or like I don’t we could put out a bunch of problems that we think might be really hard.


Prompt: But what was that famous case of that statistician who it was like, some PhD student who showed up late to a class and he saw some problems on the board and he thought they were homework, and then they were actually just open problems, and then he solved them because he thought they were homework.

Response: Right, yeah. I mean, we have much less information that these problems are hard. Again, I expect the solution to most of our problems to not be that complicated. And we’ve been working on it in some sense for a really long time. Total years of full time equivalent work across the whole team is like probably like 3 years of full time equivalent work in this area spread across a couple of people. But that’s very little compared to a problem. It is very easy to have a problem where you put in 3 years of full time equivalent work. But in fact, there’s still an approach that’s going to work quite easily with like, 3 to six months if you come at a new angle. And we’ve learned a fair amount from that that we could share, and we probably will be sharing more over the coming months.


Prompt: As far as funding goes, is this something where, I don’t know, if somebody gave you a whole bunch of money that would help? Or does it not matter how many people are working on this, by the way?

Response: So we have been right now, there’s 4 of us full time, and we’re hiring for more people.


Prompt: And then is funding that would matter?

Response: I mean, funding is always good. We’re not super funding constrained right now. The main effect of funding is it will cause me to continuously and perhaps indefinitely delay fundraising. Periodically. I’ll set out to be interested in fundraising and someone will be like, offer a grant, and then I will get to delay for another six months or fundraising or nine months, or you can you can delay the time at which Paul needs to think for some time about fundraising.


Prompt: Well, one question I think would be interesting to ask, you know, I think people can talk vaguely about the value of theoretical research and how it contributes to real world applications and you can look at historical examples or something, but you are somebody who actually has done this in a big way. Like Rlhf is something you developed and then it actually has got into an application that has been used by millions of people. Tell me about just that pipeline. How can you reliably identify theoretical problems that will matter for real world applications? Because it’s one thing to read about touring or something and the Halting problem, but here you’d have the real thing.

Response: Yeah, I mean, it is definitely exciting to have worked on a thing that has a real world impact. The main caveat I’d provide is, like, Rlhf is very simple compared to many things. And so the motivation for working on that problem was, like, look, this is how it probably should work, or this is a step in some progression. It’s unclear if it’s, like, the final step or something, but it’s a very natural thing to do that people probably should be and probably will be doing. I’m saying, if you want to talk about crazy stuff, it’s good to help make those steps happen faster, and it’s good to learn about. There’s lots of issues that occur in practice, even for things that seem very simple on paper, but mostly, like, the story of it’s just like, yeah, I think my sense of the world is things that look like good ideas on paper, just, like, often are harder than they look. But the world isn’t that far from what makes sense on paper. Like, large language models look really good on paper, and RLH looks really good on paper. And these things, I think, just work out in a way that’s yeah, I think people maybe overestimate or, like, maybe it’s kind of a trope, but people talk about, like, it’s easy to underestimate how much gap there is to practice, like, how many things will come up that don’t come up in theory. But it’s also easy to overestimate how inscrutable the world is. Like, the things that happen mostly are things that do just kind of make sense. Yeah, I feel like most ML implementation does just come down to a bunch of detail, though, of, like, build a very simple version of the system, understand what goes wrong, fix the things that go wrong, scale it up, understand what goes wrong. And I’m glad I have some experience doing that, but I think that does cause me to be better informed about what makes sense in ML and what can actually work. But I don’t think it caused me to have a whole lot of deep expertise or deep wisdom about how to close the gap.


Prompt: Yeah, but is there some tip on identifying things like Rlhf which actually do matter, versus making sure you don’t get stuck in some theoretical problem that doesn’t matter? Or is it just coincidence? Or I mean, is there something you can do in advance to make sure that the thing is useful?

Response: I don’t know if the RLHS story is, like, the best success case or something, but because the capabilities maybe I’d say more profoundly, like, again, it’s just not that hard a case. It’s a little bit unfair to be like, I’m going to predict the thing, which I pretty much think it was going to happen at some point. And so it was mostly a case of acceleration, whereas the work we’re doing right now is specifically focused on something that’s kind of crazy enough that it might not happen. Even if it’s a really good idea or challenging enough, it might not happen. But I’d say in general, and this draws a little bit on more broad experience more broadly in theory, it’s just like a lot of the times when theory fails to connect with practice. It’s just kind of clear it’s not going to connect. If you like, try if you actually think about it and you’re like, what are the key constraints in practice? Is theoretical problem we’re working on actually connected to those constraints? Is there something that is possible in theory that would actually address real world issues? I think the vast majority as a theoretical computer scientist, the vast majority of theoretical computer science has very little chance of ever affecting practice. But also it is completely clear in theory that has very little chance of affecting practice. Most of theory fails to affect practice, not because of all the stuff you don’t think of, but just because you could call it like dead on arrival, but you could also be like, it’s not really the point. It’s just like mathematicians also are like, they’re not trying to affect practice and they’re not like, why does my number theory not affect practice? It was kind of obvious. I think the biggest thing is just like, actually caring about that and then learning at least what’s basically going on in the actual systems you care about and what are actually the important constraints. And is this a real theoretical problem? The basic reason most theory doesn’t do that is just like, that’s not where the easy theoretical problems are. So I think theory is instead motivated by like, we’re going to build up the edifice of theory and sometimes there’ll be Opportunistic. Opportunistically we’ll find a case that comes close to practice, or we’ll find something practitioners are already doing and try and bring into our framework or something. But theory of change is mostly not this thing is going to make into practice. It’s mostly this is going to contribute to the body of knowledge that will slowly grow. And sometimes opportunistically yields important results.


Prompt: How big do you think a seed AI would be? What is the minimum sort of encoding of something that is as smart as a human?

Response: I think it depends a lot what substrate it gets to run on. So if you tell me how much computation does it get before or what kind of real world infrastructure does it get? You could ask what’s the shortest program, which if you run it on a million h 100s connected in a nice network with a hospitable environment will eventually go to the stars. But that seems like it’s probably on the order of tens of thousands of bytes or I don’t know if I had to guess the median, I’d guess 10,000 bytes.


Prompt: Wait, the specification or the compression of just the program?

Response: A program which went wrong. Oh, got it. But that’s going to be like, really Cheatsy. So they ask, what’s the thing that has values and will expand and roughly preserve its value? Because that thing, the 10,000 byte thing, will just lean heavily on evolution and natural selection to get there for that. Like, I don’t know, million bytes, million bytes, 100,000 bytes, something like that.


Prompt: Do you think AI lie detectors will work where you kind of just look at the activations and not find explanations in the way you were talking about with Heuristics, but literally just like, here’s what truth looks like, here’s what lies look like. Let’s just segregate the lane space and see if we can identify the two.

Response: Yeah, I think to separate the like just train a classifier to do it is a little bit complicated for a few reasons and may not work. But if you just brought them to space and say like, hey, it’s like you want to know if someone’s lying, you get to interrogate them, but also you get to rewind them arbitrarily and make a million copies of them. I do think it’s pretty hard to lie successfully. You get to look at their brain even if you don’t quite understand what’s happening. You get to rewind them a million times. You get to run all those parallel copies into gradient descent or whatever. I think there’s a pretty good chance that you can just tell if someone is lying, like a brain emulation or an AI or whatever, unless they were aggressively selected. If it’s just they are trying to lie well rather than it’s like they were selected over many generations to be excellent at lying or something, then your ML system hopefully didn’t train it a bunch to lie. And you want to be careful about whether your training scheme effectively does that. Yeah, that seems like it’s more likely than not to succeed.


Prompt: And how possible do you think it will be for us to specify human verifiable rules for reasoning such that even if the AI is super intelligent, we can’t really understand why it does certain things. We know that the way in which it arises at these conclusions is valid. Like, if it’s trying to persuade us to something, we can be like, I don’t understand all the steps, but I know that this is something that’s valid and you’re not just making shit up.

Response: That seems very hard if you wanted to be competitive with learned reasoning, it depends a little bit exactly how you set it up. But for the ambitious versions of that, let’s say it would address the alignment problem, they seem pretty unlikely, like 5% kind of thing.


Prompt: Is there an upper bound on intelligence? Not in the near term, but just like super intelligence at some point. How far do you think that can go?

Response: It seems like it’s going to depend a little bit on what is meant by intelligence. It kind of reads as a question that’s similar to is there an upper bound on strength or something? There are a lot of forms. I think it’s like the case that I think there are sort of arbitrarily smart input output functionalities and then if you hold fixed the amount of compute, there is some smartest one if you’re just like, what’s the best set of ten to the 40th operations? There’s only finitely many of them. So some best one for any particular notion of best that you have in mind? So I guess I’m just like for the unbounded question where you’re allowed to use arbitrary description complexity and compute, like probably no and for the I mean, there is some optimal conduct if you’re like I have some goal in mind and I’m just like, what action best achieves it? If you imagine like a little box embedded in the universe, I think there is kind of just like an optimal input output behavior. So I guess in that sense I think there is an upper bound, but it’s not saturatable in the physical universe because it’s definitely exponentially slow, right?


Prompt: Yeah. Because of comms or other things or heat. It just might be physically impossible to instagram something smarter than this.

Response: Yeah, I mean, like, for example, if you imagine what the best thing is, it would almost certainly involve just like simulating every possible universe. It might be in modular moral constraints, which I don’t know if you want to include like so that would be very slow. It would involve simulating like, I don’t know exactly how slow, but like double exponential very slow.


Prompt: Carl Schulman laid out his picture of the intelligence explosion in the seven hour episode. I know you guys have talked a lot. What about his basic is? Do you have some main disagreements? Is there some crux that you guys have explored?

Response: It’s related to our timelines discussion from yeah, I think the biggest issue is probably error bars where Carl has a very software focused, very fast kind of takeoff picture. And I think that is plausible, but not that likely. I think there’s a couple of ways you could perturb the situation and my guess is one of them applies. So maybe I have like I don’t know exactly what Carl’s probability is. I feel like Carl’s going to have like a 60% chance on some crazy thing that I’m only going to assign like a 20% chance to or 30% chance or something. And I think those kinds of perturbations are like one, how long a period is there of complementarity between AI capabilities and human capabilities which will tend to soften takeoff? Two, how much diminishing returns are there on software progress, such that is a broader takeoff involving scaling, electricity production and hardware production. Is that likely to happen during takeoff, where I’m more like 50 50 or more stuff like this?


Prompt: Yeah. Okay, so is it that you think the alternate constraints will be more hard? The basic case he’s laid out is that you can just have a sequence of things like flash attention or Moe, and you can just keep stacking these kinds of things on.

Response: I’m very unsure if you can keep stacking them or like it’s kind of a question of what’s like, the returns curve and Carl has some inference from historical data or some way he’d extrapolate the trend. I am more like 50 50 on whether the software only intelligence explosion is even possible, and then like a somewhat higher probability that it’s slower than why.


Prompt: Do you think it might not be possible?

Response: Well, the entire question is like, if you double R and D effort, do you get enough additional improvement to further double the efficiency? And that question will itself be a function of your hardware base, like how much hardware you have. And the question is like, at the amount of hardware we’re going to have and the level of sophistication we have as the process begins. Is it the case that each doubling of actually the initial only depends on the hardware, or like, each level of hardware will have some place at this dynamic asymptotes so the question is just like, for how long? Is it the case that each doubling of R and D at least doubles the effective output of your AI research population? And I think I have a higher probability on that. I think it’s kind of close. If you look at the Empirics, I think the Empirics benefit a lot from continuing hardware scale up so that the effective R and D stock is significantly smaller than it looks, if that makes sense.


Prompt: What are the Empirics you’re referring to?

Response: So there’s kind of two sources of evidence. One is like, looking across a bunch of industries at like, what is the general improvement with each doubling of either R and D investment or experience, where it is quite exceptional to have a field with not anyway. It’s pretty good to have a field where each time you double R and D investment, you get a doubling of efficiency. The second source of evidence is on actual algorithmic improvement in ML, which is obviously much, much scarcer. And there you can make a case that it’s been like each doubling of R and D has given you roughly a forex or something increase in computational efficiency. But there’s a question of how much that benefits. When I say the effect of R D stock is smaller, I mean we scale up. You’re doing a new task like every couple years, you’re doing a new task because you’re operating a scale much larger than the previous scale. And so a lot of your effort is how to make use of the new scale. So if you’re not increasing your installed hardware base or just flat at a level of hardware, I think you get much faster diminishing returns than people have gotten historically. I think Carl agrees, in principle, this is true. And then once you make that adjustment, I think it’s, like, very unclear where the empirics shake out. I think Carl has thought about these more than I am, so I should maybe defer more. But anyway, I’m at like 50 50 on that.


Prompt: How have your timelines changed over the last 20 years?

Response: Last 20 years?


Prompt: Yeah. How long have you been working on anything related to AI?

Response: So I started thinking about this stuff in 2010 or so. So I think my earliest timeline prediction will be in 2011. I think in 2011, my rough picture was like, we will not have insane AI in the next ten years. And then I get increasingly uncertain after that. But we converged to 1% per year or something like that. And then probably in 2016, my take was, like, we won’t have crazy AI in the next five years, but then we converged to, like, one or 2% per year after that. Then in 2019, I guess I made a round of forecasts where I gave like 30% or something to 25% to crazy Eye by 2040 and like 10% by 2030 or something like that. So I think my 2030 probability has been kind of stable, and my 2040 probability has been going up. And I would guess it’s too sticky. I guess that 40% I gave at the beginning is just, like, from not having updated recently enough, and I maybe just need to sit down. I would guess that should be even higher. I think, like 15% in 2030. I’m not feeling that bad about this is just like, each passing year is, like, a big update against 2030. We don’t have that many years left, and that’s roughly counterbalanced with AI going pretty well. Whereas for the 2040 thing, the passing years are not that big a deal. And as we see that things are basically working, that’s like, cutting out a lot of the probability of not having AI by 2040. My 2030 probability up a little bit, like, maybe twice as high as it used to be or something like that. My 2040 probability up much more significantly.


Prompt: How fast do you think we can keep building Fabs to keep up with the eye demand?

Response: Yeah, I don’t know much about any of the relevant areas. My best guess is my understanding is right now, like 5% or something of the next year’s total or best process. Fabs will be making AI hardware, of which only a small fraction will be going into very large training runs. Like, only a couple. So maybe a couple of percent of total output, and then that represents maybe like 1% of total possible output. A couple of percent of leading process 1% of total or something. I don’t know if that’s right, but I think it’s like the rough ballpark we’re in. I think things will be pretty fast. You scale up for the next order of magnitude or two from there because you’re basically just shifting over other stuff. My sense is it would be like years of delay. There’s like, multiple reasons that you expect years of delay for going past that, maybe even at that you start having. Yeah, there’s just a lot of problems. Like building new fabs is quite slow and I don’t think there’s like, TSMC is not like, planning on increases in total demand driven by AI. Like kind of conspicuously not planning on it. I don’t think anyone else is really ramping up production in anticipation think and then similarly just building data centers of that size seems like very, very hard and also probably has multiple years of delay.


Prompt: What does your portfolio look like?

Response: I’ve tried to get rid of most of the AI stuff that’s Plausibly implicated in policy work or like CEG advocacy on the RSP stuff for my involvement with Anthropic.


Prompt: What would it look like if you.

Response: Had no conflicts of interest and no inside? Like, I also still have a bunch of hardware investments which I need to think about, but I don’t know a lot of TSMC. I have a chunk of Nvidia, although I just keep betting against Nvidia constantly since 2016 or something. I’ve been destroyed on that bet. Although AMD has also done fine. The case now is even easier, but it’s similar to the case in the old days, just a very expensive company. Given the total amount of R and D investment they’ve made, they have like, whatever, a trillion dollar valuation or something that’s like very high. So the question is, how expensive is it to make a TPU? So it actually outcompetes H 100 or something. And I’m like, wow, it’s real level, high level of incompetence if Google can’t catch up fast enough to make that trillion dollar valuation not justified.


Prompt: Whereas with TSMC they have a harder remote, you think?

Response: Yeah, I think it’s a lot harder, especially if you’re in this regime where you’re trying to scale up. So if you’re unable to build fabs, I think what will take a very long time to build as many fabs as people want, the effect of that will be to bid up the price of existing fabs and existing semiconductor manufacturing equipment. And so just those hard assets will become spectacularly valuable, as will the existing GPUs and the actual yeah, I think it’s just hard. That seems like the hardest asset to scale up quickly. So it’s like the asset, if you have like a rapid run up, it’s the one that you’d expect to most benefit. Whereas Nvidia’s stuff will ultimately be replaced by either better stuff made by humans or stuff made by with AI assistance. Like the gap will close even further as you build AI systems.


Prompt: Right. Unless Nvidia is using those systems.

Response: Yeah, the point is just like anybody will so dwarf past R D and there’s like just not that much stickiness. There’s less stickiness in the future than there has been in the yeah, I don’t know. So I don’t want to not commenting for any private information just in my gut having caveatted, this is like the single bet I’ve most okay not including Nvidia in that portfolio.


Prompt: And final question, there’s a lot of schemes out there for alignment and I think just like a lot of general takes and a lot of this stuff is over my head where I think I literally it took me like weeks to understand the mechanistic anomaly stuff you work on without spending weeks.

Response: How do you detect bullshit?


Prompt: People have explained their schemes to me and I’m like, honestly, I don’t know if it makes sense or not with you. I’m just like I trust Paul enough that I think there’s probably something here if I try to understand this enough. But how do you detect bullshit?

Response: Yeah, so I think it’s depends on the kind of work. So for the kind of stuff we’re doing, my guess is like most people there’s just not really a way you’re going to tell whether it’s bullshit. So I think it’s important that we don’t spend that much money on the people we want to hire are probably going to dig in in depth. I don’t think there’s a way you can tell whether it’s bullshit without either spending a lot of effort or leaning on deference with empirical work. It’s interesting in that you do have some signals of the quality of work. You can be like, does it work in practice? Does the story? I think the stories are just radically simpler and so you probably can evaluate those stories on their face. And then you mostly come down to these questions of like, what are the key difficulties? Yeah, I tend to be optimistic when people dismiss something because this doesn’t deal with a key difficulty or this runs into the following insurable obstacle. I tend to be a little bit more skeptical about those arguments and tend to think, like, yeah, something can be bullshit because it’s not addressing a real problem that’s I think the easiest way this is a problem someone’s interested in that’s just not actually an important problem, and there’s no story about why it’s going to become an important problem. E g, like it’s not a problem now and won’t get worse or it is maybe a problem now, but it’s clearly getting better. That’s like one way and then conditioned on passing that bar, like dealing with something that actually engages with important parts of the argument for concern and then actually making sense empirically. So I think most work is anchored by source of feedback is like actually engaging with real models. So it’s like, does it make sense how to engage with real models? And does the story about how it deals with key difficulties actually make sense? I’m pretty liberal past there. I think it’s really hard to, like, eg. People look at mechanistic, interpretability and be like, well, this obviously can’t succeed. And I’m like, I don’t know. How can you tell? It obviously can’t succeed. I think it’s reasonable to take total investment in the field. How fast is it making progress? How does that pencil I think most things people work on, though, actually pencil pretty fine. They look like they could be reasonable investments. Things are not, like, super out of whack.


Prompt: Okay, great. This is, I think, a good place to close. Paul, thank you so much for your time.

Response: Yeah, thanks for having me. It was good chatting.