Prompt: What kinds of things should people do before we have an artificial and general intelligence in order to, I guess, protect themselves financially, if they’re potentially going to lose their jobs? Is there really anything meaningful that people can do to shield themselves from potentially negative effects?

Response: If the world continues to go, well … If all that happens is that we build AI, and it just works the way that it would work in an efficient market worlds, there’s no crazy turbulence, then the main change is, you shift from having … Currently two-thirds of GDP gets paid out roughly as income. I think if you have a transition too human labor being obsolete then you fall to roughly zero of GDP is paid as income, and all of it is paid out as returns on capital. From the perspective of a normal person, you either want to be benefiting from capital indirectly, like living in a state that uses capital to fund redistribution, or you just wanna have some savings. There’s a question of how you’d wanna … The market is not really anticipating AI being a huge thing over 10 or 20 years, so you might wanna further hedge and say … If you thought this was pretty likely, then you may want to take a bet against the market and say invest in stuff that’s gonna be valuable in those cases.I think that, mostly, the very naive guess is not a crazy guess for how to do that. Investing more in tech companies. I am pretty optimistic about investing in semiconductor companies. Chip companies seem reasonable. A bunch of stuff that’s complimentary to AI is going to become valuable, so natural resources bid up. In an efficient market world, the price of natural resources is one of the main things that benefits. As you make human labor really cheap, you just become limited on resources a lot. People who own Amazon presumably benefits a huge amount. People who run logistics, people who run manufacturing, etc. I think that generally just owning capital seems pretty good. Unfortunately, right now is not a great time to be investing, but still, I think that’s not a dominant consideration when determining how much you should save.


Prompt: You say it’s bad just because the stock market in general looks overvalued, based on price-to-earnings ratios?

Response: Yeah, it’s hard to know what overvalued means exactly, but certainly it seems reasonable to think of it in terms of, if you buy a dollar of stocks, how much earnings are there to go around for that dollar of stocks, and it’s pretty low, pretty usually low. This might be how it is forever. I guess if you have the kind of you that I’m describing, if you think we’re gonna move to an economy that’s growing extremely rapidly, then you have to bet that the rate of return on capital is gonna go up, and so it’s kind of… In some sense, you need to invest early because you wanna actually be owning physical assets, since that’s where all of the value is going to accrue. But. it’s also a bummer to lock in relatively low rates of return.


Prompt: In the normal scenario, where that doesn’t happen?

Response: No, in the … Even in … Suppose I make someone a loan. A way people normally hold capital would be making a loan … You make a loan now, you make a loan at 1% real interest for 20 years. You’re pretty bummed if then people develop AI, and now the economy is growing by 25% a year. Your 1% a year loan is looking pretty crappy.And you’re pretty unhappy about that. Stocks are a little bit better than that, but it depends a lot on … Yeah, stocks still take a little bit of a beating from that. I think this generally is a consideration that undercuts the basic … I think the basic thing you would do if you expected AI would be save more, earn more capital if you can. I think that’s undercut a little bit by the market being priced such that it’s hard, which could be a bunch of people doing that, if that’s not why it’s happening. Prices aren’t being bid up because everyone is reasoning in this way. Prices are being bid up just ’cause of unrelated cyclical factors.


Prompt: Let’s talk now about some of the actual technical ideas you’ve had for how to make machine learning safer. One of those has been called iterated intelligence distillation and amplification, sometimes abbreviated as IDA. What is that idea in a nutshell?

Response: I think the starting point is realizing that it is easier to train an AI system, or it currently seems easier to train in an aligned AI system, if you have access to some kind of overseer that’s smarter than the AI you’re trying to train. A lot of the traditional arguments about why alignment is really hard, or why the problem might be intractably difficult, really push on the fact that you’re trying to train, say, a superintelligence, and you’re just a human.And similarly, if you look at existing techniques, if you look at the kind of work people are currently doing in more mainstream alignment work, it’s often implicitly predicated on the assumption that there’s a human who can understand what the AI is doing, or there’s a human who can behave close to approximately rational, or a human who can evaluate how good the AI system’s behavior is, or a human who can peer in at what the AI system is thinking and make of that decision process.And sometimes its dependence is a little bit subtle, but I think, it seems to me like it’s extremely common. Even when people aren’t acknowledging explicitly a lot of the techniques are gonna have a hard time scaling to domains where the AI is a lot smarter than the overseer who’s training it. I think motivated by that observation you could say, let’s try and split the alignment problem into two parts, one of which is try and train an aligned AI, assuming that you have an overseer smarter than that AI, and the second part is actually produce an overseer who’s smart enough to use that process or smart enough to train that AI.The idea in iterative amplification is to start from a weak AI. At the beginning of training you can use a human. A human is smarter than your AI, so they can train the system. As the AI acquires capabilities that are comparable to those of a human. Then the human can use the AI that they’re currently training as an assistant, to help them act as a more competent overseer.Over the course of training, you have this AI that’s getting more and more competent, the human at every point in time uses several copies of the current AI as assistants, to help them make smarter decisions. And the hope is that that process both preserves alignment and allows this overseer to always be smarter than the AI they’re trying to train. And so the key steps of the analysis there are both solving this problem, the first problem I mentioned of training an AI when you have a smarter overseer, and then actually analyzing the behavior of the system, consisting of a human plus several copies of the current AI acting as assistants to the human to help them make good decisions.In particular, as you move along the training, by the end of training, the human’s role becomes kind of minimal, like if we imagine training superintelligence. In that regime, we’re just saying, can you somehow put together several copies of your current AI to act as the overseer? You have this AI trying to … Hopefully at each step it remains aligned. You put together a few copies of the AI to act as an overseer for itself.


Prompt: How is it that the kind of training AI and the human are gonna be smarter than the AI that they’re trying to train? I mean they’re trying to make something that’s smarter than them, right? But you’re saying that as a … That in combination, they can do a better job?

Response: Yeah, so the key step there, the reason why it’s plausible … Let’s suppose we’re trying to train our AI to some level of capability. Suppose that you imagine the AI gets to think for two seconds before each decision it makes. Then when, in this oversight process, we have multiple copies of the AI, each thinking for two seconds. If you imagine this in an analogy to humans, you have one smart human is making a decision, and they’re being overseen by a team of 20 slightly less-smart humans. And when I say slightly, I really mean slightly, like they’re one gradient step behind. The hope is, and this is a complicated question or an open question at the moment … If you have this team of 20 very slightly dumber aligned agents, then can they divide up the work of overseeing this AI, so that even they’re a tiny bit dumber, by working together, they can do it.


Prompt: Okay, so is the idea that by throwing more processing power or more instances behind the somewhat dumber algorithm, they’ll be smarter than the AI that they’re trying to train, and therefore able to keep it under control and give it good instructions in concert with the human?

Response: That’s roughly right, yeah. In particular, when we ask, what does this team have to do? We have this overseer, which is now a team of AIs working together. What does the team have to actually do? There may be two big classes of tasks they have to do. One is, they’re gonna provide the incentives to the AI. This team is gonna look, or AI is gonna take some action, then our team of overseer AIs is going to evaluate that action and decide how good it was, and that’s gonna be the rewards signal that we use to train our AI.In order for that to work, this team of 20 AIs needs to be able to make a better judgment about how good actions are than the AI that they’re trying to train. Right? Because they can’t … They’re sort of using this process. Can’t really train an AI to make smarter decisions than that team would make. And so, there, we hope that you can sort of break down … There’s this question, how good an action was action x? You hope that you can break down the problem of evaluating that action into a bunch of tasks, each of which are a little bit easier, and then you can split up those tasks across this team. That’s one of the tasks that the team needs to be able to do, this evaluation. Answer these evaluation questions.The second category that we don’t understand as well, which is like, they would like to ensure that the AI they’re training reliably does the thing they’re trying to incentivize it to do. For example, an example of a technique called adversarial training that you could use for this problem is, if I’m training some AI, and I’m concerned that in some situations it might behave badly, I can try and synthesize situations. I can try and construct hand-crafted situations, designed to look like situations where the AI might behave badly. And I can, instead of running, say, just on real examples, I also run it on these synthetic examples designed to cause problems.And so in order to that, I need to be able to do tasks like look at the AI and figure out whether the synthetic, what are the cases where it would perform poorly, and then also evaluate in such a case, did it do something unacceptable? I think it’s not crazy to think about this with an analogy to humans. Again, you have your team of 20 humans. You have one slightly smarter human, who’s maybe is … You’re concerned that under some conditions, they’re gonna try and break this process. They’re gonna try and go rogue, do something bad. And this team of 20 humans is responsible for interrogating this human and making sure there’s not a condition under which it’s going to behave poorly. Giving it test cases that hopefully it can’t tell, the human being trained can’t tell, from the real situations that it’s going to encounter.That’s an example of a technique that you could use for this process of reliability. There are other possible techniques you could use. And those are the two big tasks that this team of AIs needs to be able to solve. It was a team of humans in the analogy, but … This team needs to be able to evaluate behavior to produce good incentives, to train the AI to do what the team wants it to do, and it needs to be able to do some evaluation for reliability, which is not something we currently understand well.


Prompt: At its core, you’re going to try to get somewhat dumber AIs and humans together to come up with a training process by which they figure out whether this smarter AI that they’re trying to develop is behaving in the way that they want, by designing particular scenarios to test whether that’s the case. And even though they’re not quite as smart, I guess, in this model, because you’re throwing quite a lot of power behind that somewhat simpler task of just evaluating whether it’s doing the right thing, you hope that that way you’ll be able to gradually scale up and not lose alignment at any particular point in time.

Response: Yeah, that’s right. I guess it’s worth pointing out again that the … Generating scenarios, that’s one possible way of testing, trying to get the system to behave robustly, robustly to do the right thing. There are other possible approaches. You could also try and use an approach where you open up the brain of this AI you’re trying to train. Use something like interpretability techniques that people are currently working on, to understand how it’s thinking, and say, ah, now that I understand how it’s thinking, I see that here’s a place that it’s thinking that I wouldn’t want it to be thinking. And I can tell from that that it will fail in the scenario. Or I can just directly say, no, that was not a good way to be thinking about the problem, and penalize that. One of the major things this group is doing is just determining incentives for the AI that they’re training. This team of slightly dumber humans is just determining what … They’re evaluating the AI on realistic examples, on examples that appear in the real world and saying, how good was its behavior in this case? How good was its behavior in that case? And the AI is being trained to maximize those evaluations.


Prompt: By “incentives,” you mean, do we give it its reward? Do we give it whatever it’s been programmed to try to get?

Response: Yeah. I mean, formally, you would really be using gradient descents, where you’re like, yup, we take our AI, we take this evaluation that this team is providing, and then we modify the AI very slightly, so that it gets a slightly higher reward on that, a slightly higher evaluation, or it outputs actions that have higher evaluations on average. And in that setting, actually the AI that you’re starting with is exactly the same as the AIs who are on this team doing the oversight. But after you make this very small perturbation, that perturbation now hopefully gives you an AI that’s very slightly smarter than the other AIs on the team. The AI that’s actually thinking is exactly as smart as the ones on the team. It’s only as you consider these possible perturbations that you hope that the perturbations are like epsilon smarter. And that’s how training would normally work, were you’d have some evaluation, consider AI, run it, perturb it to get slightly better performance, repeat.


Prompt: Someone emailed me about IDA wanting me to ask you about it and said, “The context here is that I and many others think that IDA is currently the most promising approach to solving the alignment problem, largely because it’s the only real, actual proposal that anyone has made.” Do you think that’s right? And, more generally, what’s been the reaction to this general approach?

Response: Yeah. I would say the current situation is, I am very interested in really asking what solutions would look like in … as you scale them up. What is our actual game plan? What is the actual end-game here? That’s a question that relatively few people are interested in, and so very few people are working on. MIRI, the Machine Intelligence Research Institute, is very interested in that question, but they part ways with me by believing that that question is so obviously impossible that it’s not worth thinking about it directly, and instead we should be trying to improve our understanding of the nature of rational agency. That’s the reason, to me, why they are not in the business of trying to produce concrete proposals. It just seems doomed to them. Feels to them like we’re just patching holes in a thing that’s fundamentally not going to work.And most people in the broader ML community, I would say they take an attitude that’s more like, we don’t really know how the system is going to work until we build it. It’s not that valuable to think about in advance, what is the actual scheme going to look like. And that’s the difference there. I think that’s also true for many safety researchers who are like most, who are more traditional AI or ML researchers. They would more often say, look, I have a general plan. I’m not going to think in great detail about what that plan is going to look like because I don’t think that thinking is productive, but I’m gonna try and vaguely explain the intuitions, like maybe something like this could work. I think it sort of happens to be the case. Basically no one is engaged in the project of actually, say, here is what aligned AI might look like.I’m trying to aspire to the goal of actually write down a scheme that could work. There are a few other groups that are also doing that. I guess, at the OpenAI safety team, we also recently published this paper on safety via debate, which I think also has this form of being also an actual candidate solution, or something that’s aspiring to be a scalable solution to this problem. Jeffrey Irving was lead author on that. He’s a colleague on the OpenAI safety team. I think that’s coming from a very similar place. And maybe is, in some sense is a very similar proposal.I think it’s very likely that either both of these proposals work, or neither of them works. In that sense they’re not really totally independent proposals. But they’re getting at … They’re really pushing on the same facts about the world that let you make AI. Both of then are leveraging AI to help you evaluate your AI.I think the other big category is work on inverse reinforcement learning, where people are attempting to invert through human behavior and say, given what a human did, here’s what the human wants. Given what the human wants, we can come up with better plans to get what the human wants, and maybe that approach can be scalable. I think the current state of affairs on that is there are some very fundamental problems with making it work, with scaling it up, related to, how do you define what it is that a human wants? How do you relate human behavior to human preferences, given that humans aren’t really the kind of creature that actually has … There’s no slot in the human brain that’s where you put the preferences.I think unfortunately we haven’t made super much progress on that core of the problem, or what I would consider the core of the problem. I think that’s related to people in that area not thinking of that as being their current, primary goal. That is, they’re not really in the business of saying, and here we’re gonna try and write down something that’s just gonna work, no matter how powerful AI gets. They’re more in the business of saying, let’s understand. Clarify the nature of the problem, make some progress, try and get some intuition for what will alow us to make further progress, and how we could get ourselves in a position where, as AI improves, we’ll be able to adapt to that.I think it’s not a crazy perspective. But I think that’s how we come to be in this place where there are very, very few concrete proposals that are aspiring to be actual … a scheme you could write down and then run with AI and would actually [yield 01:33:10] AI. I think overall reaction is there’s two kind of criticisms people have, one of which is, this problem seems just completely hopeless. There’s a few reasons people would think that this iterative amplification approach is completely hopeless. They’re normally can be divided roughly into thinking that organizing a team of 20 AIs to be aligned and smarter than the individual AIs already subsumes the entire alignment problem. In order to do that, you would need to understand how to solve alignment in a very deep way, such that if you understood that, there’d be no need to do any of this, bother with any of the other machinery.The second common concern is that this robustness problem is just impossibly difficult. In the iterative amplification scheme, as we produce a new AI, we need to verify … Not only do we need to incentivize the AI to do well on the training distribution. We also need to sort of restrict it to not behave really badly off of the training distribution. And there are a bunch of plausible approaches to that that people are currently exploring in the machine learning community. But it’s … I think the current situation is, we don’t see a fundamental reason that’s impossible, but it currently looks really hard. And so many people are suspicious that problem might be impossible.That’s one kind of negative response is this … Maybe the problem, iterative amplification, cannot be made to work. The other kind of response is, it’s reasonably likely that AI safety is easy enough that we also don’t need any of this machinery. That is, I’ve described this procedure for trying to oversee AI systems that are significantly smarter than humans. Many of the problems on this perspective are only problems when you want things to be scalable to very, very smart AI systems. You might think, instead, look, we just want to build an AI that can take one “pivotal act,” that is an expression people sometimes use for an action an AI could take that would substantially improve our situation, with respect to the alignment problem. Say we want to build an AI which his able to safely take pivotal act. That doesn’t require being radically smarter than a human or taking actions that are not understandable to a human, so we should really not be focusing on or thinking that much about techniques that work in this weird, extreme regime.I guess even people in the broader ML community would say, look, I don’t know … They don’t necessarily buy into this framework of, can you just take a pivotal act? But they would still say, look, you’re worrying about a problem which is quite distant, it’s pretty likely that for one reason or another that problem is going to be very easy by the time we get there, or that one of these other approaches we can identify is just going to turn out to work fine. I think both those reactions are quite common. I think there’s also a reasonably big crowd of people who are like, yeah, I’m really interested, coming from a similar perspective to me, where they really want a concrete proposal that they can actually see how it could work. I think that those people tend to be, well, for those who aren’t incredibly pessimistic about this proposal, many of them are pretty optimistic about iterative amplification, or debate, or something along those lines.


Prompt: That’s a great answer. Yeah, I think it’s really creditable that you actually try to put out ideas for how we could deal with this, and I’ve seen, as you said, very few other people actually try to do that. And people can just read those ideas for themselves on your AI Alignment blog on Medium. You mentioned another approach that people have been talking about recently, which is debate as a way of aligning AI. You also mentioned inverse reinforcement learning. But we discussed that in the episode last year with Dario Amodei, so we’ll skip that one. But can you just describe the approach in the debate paper, which is somewhat similar, it sounds like, to IDA?

Response: Yeah. The idea is, we’re interested in training AI systems to make decisions that are in some respects too complicated for a human to understand. It’s worth pointing out that problems can appear probably long before AI is broadly-human level, because AI’s abilities are very uneven, so it can have an understanding of a domain that is way beyond human understanding of that domain, even while being subhuman in many respects. We want to train this AI to make decisions that are too complex for a human to understand. We’re wondering how do you get a training signal for such an AI? One way, one approach people often take is to pick some actual consequence in the world, like some simple consequence in the world that you could optimize, like whatever, you’re running a company, just … I don’t care how you’re making decisions about that company. All I care about is that they lead to the company having high profit.We’re interested in moving away from … I think there are serious concerns with that, from a safety perspective. We want to move more towards the regime where, instead of evaluating, yes, this decision had good consequences, but I don’t understand why, we’re evaluating a proposed decision and saying, yeah, we understand that that’s a good decision, so we’re going to give it a high reward because we understand why it’s good. That approach has … I mean, if an AI comes to you and says, “I would like to design the particle accelerator this way because,” and then makes to you an inscrutable argument about physics, you’re faced with this tough choice. You can either sign off on that decision and see if it has good consequences, or you can be like, no, don’t do that ’cause I don’t understand it. But then you’re going to be permanently foreclosing some large space of possible things your AI could do.Instead, the proposal is, we’re going to have two AIs. One AI’s gonna make a proposal. We can’t directly … That proposal counts on a bunch of complicated facts that we don’t necessarily understand. It’s gonna make some complicated argument about the economy in order to justify that proposal. And we couldn’t actually evaluate that argument. But if we introduce this adversarial agent who can explain to us why the proposal that was made is bad, and the original agent, if this critique has a flaw, the original can say, no, that critique is not a valid critique because, and point out the flaw. And then the critiquer can say, no, actually it was valid. They can go back and forth in this way.Then you can implicitly explore an exponentially large space of considerations. Because by giving the critiquer the option to pick any line of argument that they want in order to attack the proposal, you can verify that every possible line of argument, if the critiquer is not able to win, it suggests to you that every possible line of argument would have been unsuccessful. Every possible line of argument would have still left you thinking the proposal was a good one. It’s not clear if you can actually construct.Now we have some complicated question. Our AI is proposing to you an action. We would like to set up the debate such that the best action will actually win the debate. If two AIs propose actions, and one of them is proposing an action which is actually better, then it will be able to win a debate in which it establishes that its action is better.I think there are some plausibility arguments, like the one I just gave, that you’re exploring an exponentially large space of considerations. But this might be possible in cases where a human couldn’t have any idea about the task itself, or directly answering the question. It’s a very open question, exactly how powerful is debate? That is, if we set up a debate in the best possible way, so we give it, we have some human judge of this debate who’s evaluating the claims and counter-claims. If we give them optimal training and optimal advice, and then we have two very powerful agents debate in this way, we’d like it to be the case that the optimal strategy in this debate is being honest and actually telling the truth and then actually providing valid arguments for that and responding to counterarguments in a valid way. And we don’t know if that’s the case, but figuring out if that’s the case and then understanding in what cases we’re able to run such debates and it converges to truth, understanding how to set them up so they converge to truth, etc., does give a plausible way of training powerful AI systems.


Prompt: So how analogous is this approach to a case where say a person like me is trying to judge a difficult scientific issue, and I wouldn’t be capable of doing the original research and figuring out the truth for myself, but if there was scientists debating back and forth and one of them maybe was trying to be misleading in some way and another one was being truthful, the hope is that I would be able to figure out which one was telling the truth because I can at least evaluate the debate even if I couldn’t produce the arguments, myself?

Response: Yeah, so I think the situation’s pretty analogous to two human experts with lots of understanding you lack; you’re trying to understand the truth. You hope that if one of those experts is trying to make a claim that is true, then by zooming in on one consideration after another, you could find out if it’s true. You could eventually come to be very skeptical all the counterarguments or they could undermine all the counterarguments that were offered, and so I think that’s like … it’s definitely not an obvious claim. It’s not obvious in the context of human discussions. I think as a society, empirically, there aren’t great examples of covering really big gaps in expertise. Like, it’s often the case that two people with expertise in their area can have a debate in a way that convinces someone with slightly less expertise, but when there’s really large gaps, I don’t think we have a very good record of doing that kind of thing successfully.So, I’d say there’s more hope that this is possible, that a human could just evaluate some proposal produced by a sophisticated AI system, but it’s still very much an open question whether this kind of thing could actually work and one way you could try and assess that would be say, “We’re gonna get fairly serious about … have some serious experiments of trying to take people with considerable expertise in an area, have them have a debate arbitrated by someone with less expertise.


Prompt: What do you think is the biggest or most concerning criticism of AI safety via debate?

Response: Personally, I think the worst problem is just, is the basic question, do debates tend to favor accurate answers, or do they tend to favor answers that are easy to defend for reasons other than their accuracy? There’s a bunch of reasons the debate might favor an answer other than it being accurate. I think one that really leaps to people’s mind is, well, the judge is just a human. Humans have all sorts of biases and inconsistencies. That’s one reason that debate could favor answers other than the accurate one. I’m more personally concerned about maybe an even more basic question, which is, setting aside all human biases and all ways in which humans fail to reason well, I think it’s just an open question: does the structure of debate tend to promote truth? Does it tend to be the case that there’s some way to argue for the accurate position, even if the content of the debate, the thing you’re debating, is really, really complex compared to what the human can understand?


Prompt: It seems like debate among humans is way better than random, anyway.

Response: I agree. Humans are … And clearly we’re able to get, at least in some cases, able to get much better answers than we’d get on our own. If I get to observe two experts debate a subject, even if one of them is actively trying to mislead me, I can arrive at better conclusion than I could have arrived at if I just wasn’t able to listen to their expertise or was only given a single expert, whose incentive was to look good.I think that the example of debates amongst humans makes it very unclear whether this procedure can be scaled arbitrarily far. An example you might think of is, consider a human who’s a smart person, who knows a lot about reason, who has been practiced a lot at judging debates. They don’t know any calculus. They’re not judging a debate between two quantum physicists about how to determine the results of some experiment on a recent particle accelerator. Just imagining that process, I can see how it could work, I can imagine it working, but it’s an incredibly intimidating prospect. This person is not gonna understand anything about … Over the course of the debate, there’s no way that they can form in their head a reasonable model of calculus, or of quantum mechanics, or of the standard model. And yet you hope that somehow the way they’re arbitrating this debate can implicitly answer extremely complex questions about … that depend on all those areas.I think this is the kind of test that you can do empirically. We can ask the empirical question, for a human who’s very good at, a very smart person who’s been trained to judge such debates, and then you have two people with a ton of expertise in an area they’ve never thought about come in, one of them trying to convince them of the truth and one trying to mislead them, is it the case empirically that humans can arbitrate such debates and actually, that the best way to win such a debate is to provide true facts about the domain to the human?I think if that’s the case, I think it’s actually, if that’s the case, it’s a very interesting fact, not just for the purpose of AI but just in general. I think it, prima facie, is just a really important question about the world. Are there norms of debate that allow you to consistently arrive at the truth in domains where the arbitrator doesn’t understand what’s true? That’s a question that’s relevant to a ton of domains.This version of the question is distinctive in some respects. I think mostly it’s distinctive because we are free to set things up in the way that’s maximally convenient. It’s kind of asking, under the best possible conditions, can debate be conducive to truth? Whereas most debates in our current society are under pretty highly suboptimal conditions, like very limited time, bad incentives on behalf of the judge, judges sampled from some population that doesn’t have a lot of time to think about how to judge such debates well, or isn’t that … hasn’t thought a lot about how to structure this to lead to truth. I think most debates in the real world are under pretty pessimistic conditions, but just understanding when does debate work, or what is the equivalent of debate truth, I think is a really … I would consider that a really fundamental and interesting question, completely independent of AI. I think it’s now also a particularly important question because it really is closely related to one of the most plausible strategies for training very powerful AIs to help us actually arrive at good advice or good conclusions.


Prompt: Are there other important pros and cons of this approach that are worth mentioning?

Response: So, I think there’s definitely a lot that could be said about it. There are a bunch of other issues that come up like when you start actually trying to do machine learnings, when you try and train agents to play this kind of game, then there’s lots of ways that that can be hard as a machine learning problem. You can have lots of concerns in particular with the dynamics of this game. So, some people maybe wouldn’t be happy that you’re training your AIs to be really persuasive to people. You might be concerned that makes some kind of familiar modes look more … crop up in more subtle ways or be more problematic.But, I really think the main thing is just: is it the case that a sufficiently sophisticated judge will be able … every judge defines a different game, like convincing me is a different game from convincing you. I think it’s clear that for weak enough judges, this game isn’t particularly truth conducive. There’s no reason that the honest player would have an advantage. The hope is that there is some level of sufficiently strong judges, above which, it’s the case that you converge over longer and longer debates to more accurate claims, yeah, it’s unclear. So, first question is: is there a threshold, and the second question: are humans actually above that threshold? If this was the case, if we have humans judge such debates, they will actually have honest strategies winning.


Prompt: What kind of people do you need to pursue this research? Are there any differences compared with other lines?

Response: So, again, I think there’s like a very similar … there’s a bunch of different questions that come up both for amplification and debate. I think different questions require different kinds of skill and different backgrounds. I think that both for amplification and debate, there is this more conceptual question. Or, I don’t know if conceptual is the right word. It’s a fact both about like the structure of argument and about the actual way humans make decisions, which is like, “Can humans arbitrate these debates and demands where they lack expertise? Or, in the amplification case, can you have teams addressing some issue where no individual can understand the big picture?And that, I mean, there’s a bunch of different angles you could take on that question. So, you could take a more philosophical angle and say, “What is going on there? Why should we expect this to be true, or what are the cases that might be really hard?” You could also just run experiments involving people, which seems relatively promising, but involves, obviously, a different set of skills, or you could try and study it in the context of machine learning and go straight for training … you might say, “Well, we could test these things with humans if we had very large numbers of humans.” Maybe, actually, the easiest way to test it is to be in the regime where we can train machines to effectively do things that are much, much more expensive than what we could afford to do with humans.So, you could imagine approaching it from a more philosophical perspective, a more, I don’t know, cognitive science or just organizing, maybe not even in an academic perspective, just putting together a bunch of humans and understanding how those groups behave, or a more machine learning perspective.


Prompt: What’s been the reception from other groups to this debate approach?

Response: So, I think there are many different groups and different answers for different groups. I would say that for many people, the nature of the problem or the alignment problem is very unclear when they first hear it stated in an abstract way, and so I think for a lot of people, it’s been very helpful to get a clear sense of what the problem is that you’re trying to solve. I think when you throw out this proposal, people both understand why debates are better than just giving an answer and having a human evaluate it, and they also can sort of see very clearly why there’s difficulty, like it’s not obvious that the outcome of the debate is in fact producing the right answer.So, I think from that perspective, it’s been extremely helpful. I think a lot of people have been able to get much more purchase understanding what the difficulties are in what we’re trying to do. I think for people who are more in like the ML side … again, it’s still been very helpful for having them understand what we’re trying to do, but I think the ML community is really very focused on a certain kind of implementation and actually building the thing, and so I think that community is mostly just sort of waiting til, “That’s a very interesting research direction,” and then their response is to wait until things either happen or don’t happen, til we’ve actually built systems that embody those principals to do something which you wouldn’t have been able to do without that idea.


Prompt: So, if we can use this approach to go from having like 60% accuracy to 70% or 80%, how useful is that? Do we need to be able to judge these things correctly almost all of the time, or is it more just like: the more often humans can make the right call, the better.

Response: Yeah, so, certainly, if you just had a judge who was like correct, but then 40% of the time, they err randomly, that would be totally fine. I guess that’s sort of gonna average out and it’s not a problem at all. What you really care about is just: in what cases are there, to what extent are there systematic biases in these judgements? So, to the extent that we consistently just make the wrong answer when the answer depends on certain considerations or in certain domains, and so, from that perspective, I guess the question is, “What class or problems can you successfully resolve with this technique, and if you push that frontier of problems a little bit further, you can solve a few more problems now than you could before. Are you happy?”I’d say there’s kind of two attitudes you could have on this. So, one: I guess the thing I really would like is a solution that just works in the sense that we have some principal reason to think it works. It works empirically. As we scale up our machine learning systems, it works better and better, and we don’t expect that to break down. That would be really great and sort of has, from a more theoretical perspective, that’s kind of what we’d like. There’s a second perspective you could have, which is just, “There’s this set of important problems that we want to apply machine learning systems to, so as we deploy ML systems, we think the world might change faster or become more complex in certain respects,” and what we really care about is whether we can apply machine learning to help us make sense of that kind of world or steer that world in a good direction, and so from that perspective, it’s more like there’s this list of tasks that we’re interested in and sort of the more tasks we can apply ML to, the better position we will be to cope with possible disruption caused by ML.And so, from that perspective, I think you’re just sort of happy every time you expand the frontier of tasks that you’re able to solve effectively. I think I also take that pretty seriously, so if it was the case that we could just push the set of tasks that we’re able to solve a little bit, I think that would improve our chances of coping with things well a little bit, but my main goal is, or the main focus, I think, as we are further away, if we’re having to think about things more conceptually or more theoretically, then I think it’s better to focus on having a really solid solution that we think will work all the time. As we get closer, then it becomes more interesting to say, “Great, now we see these particular problems that we wanna solve. Let’s just see if we can push our techniques a little bit so that ML systems can help us solve those problems.”


Prompt: Do you think it’s possible that there’s an advantage to whoever’s trying to be deceptive in these cases, that in fact it’s easier for the person who’s trying to mislead or judge because they can choose from a wider range of possible claims that they could make whereas the person or the agent that’s trying to communicate the truth, they can only make one claim, which is the true one?

Response: Yeah, I guess a few points: maybe a first preliminary point is that, in general, if you had two agents, there wouldn’t be one assigned to the truth and one assigned to lie. Instead, they would both just be arguing whatever they thought would be most likely to be judged as honest and helpful, so in a world where it worked like that, there would just … neither participant in the debate would be trying to say anything true. Both of them would be arguing for some garbage if we were in this unfortunate situation.So then, in terms of the actual question, yeah, you could sort of imagine there is this giant space of things you could argue for. One of them is like … some tiny space of other things that we would actually regard on reflection as the most truthy and all the other stuff, yeah it’s a very, very tiny subset of the possible claims, and there’s a ton of other things that differ between different claims besides how actually useful are they and truthful are they?And so, I think, a priori, you would definitely be … it’s like a very, very surprising claim or very, very special claim to say the very best strategy from amongst all these strategies is the one that’s most truthful and helpful, so I definitely think your first guess, just if you didn’t know anything about the domain would be that there’s going to be some other properties. Maybe how nice it sounds is very useful like you wanna pick the thing that sounds nicest, or the thing that has the slickest soundbite in its favor or something like that.I think I am reasonably optimistic that if, say, a human judge is careful, they can sort of judge well enough that they have some … I’d say, if you’re a weak judge, this process can’t really get off the ground. You’re not able to at all correlate your judgements with truth. As you get to be a stronger judge, you hope that not only can you start to answer some questions; you can sort of bootstrap up to answering more and more complex questions. So, that is, you could say, “Well, if I just were to guess something off the top of my head, that has one level of correlation with truth.” Like, in easy enough cases, that’s going to be truthful. Then, if I have a short debate that sort of bottoms out with me guessing something off the top of my head, that can be a little bit more conducive to the truth. And now, if I have a long debate where after that long debate I can now have a short debate to decide which side I think wins, then I think that’s more likely to be conducive to truth.So, you would hope that you have sort of eliminating behavior where as you think longer and longer, the class of cases in which truthfulness becomes the optimal strategy grows, but yeah, I think it’s not obvious at all.


Prompt: What’s the best source for people who want to learn more about this approach? There’s a paper up on Archive, and I think also a blog post that came out after that that’s perhaps more extensive?

Response: I think the paper’s probably the best thing to look at. So, there’s a paper in the archive called AI Safety Via Debate. Like, it covers a lot of considerations and raises a lot of considerations, discusses possible problems, discusses how it compares to amplification, things like that. It presents some very simple toy experiments to show a little bit about how this might work in the context of machine learning. It doesn’t present sort of convincing example of a system which does something something interesting using debate, and so that’s what we’re currently … that’s what we’re currently working on, and so a reader who’s looking for that should maybe come back in six months. But, I think if you want to understand what is … yeah, if you want to understand why we’re interested in the idea or what is basically going on, then I think the paper’s a good thing to look at.Advances needed to implement these strategies


Prompt: What would prevent us from implementing either of these strategies today? What advances do we need to actually be able to put them into practice?

Response: I think, depending on your perspective either unfortunately or fortunately, there’s really a ton of stuff that needs to be done. One category is just building up that basic engineering competence to make these things work at scale. In writing this process, it’s kind of like training an AI. Let’s consider the debate case, which I think is fairly similar in technical requirements, but may be a bit easier to talk about.We understand a lot about how to train AIs to play games well because that’s a thing we’ve been trying to do a lot. This, as an example of a game, has many differences from the games people normally train AIs to play. For example, it is arbitrated by a human, and queries to a human judge are incredibly expensive. That presents you with a ton of problems about, one, organizing the collection of this data. Using approximations. There’s this whole family of approximations you’re going to have to use in order to be able to actually train these AIs to play this game well. You can’t just have, every time they play the game, a human actually makes the evaluation. You need to be training models to approximate humans. You need to be using less-trusted evaluations. You need to be learning cleverly from data that’s passive data rather than actually allowing them to query the judge. That’s one …Technically, running this project at scale is hard for a bunch of reasons that AI is hard, and then also hard for some additional reasons distinctive to the role of humans in these kinds of proposals. It’s also hard, I guess, as a game, because it has some features that games don’t normally have. So we’re used to thinking of games with … there’s other technical differences beyond the involvement of humans that make these kind of hard engineering problems. And some of those are things that I’m currently working on, just trying to understand better. And, again, trying to build up the actual engineer expertise to be ready to make these things work at very large scale. So that’s one class of problems.A second class of problems is just figuring out … I think there’s maybe two things you could want. One is you want to be able to actually apply these schemes. You want to be able to actually run such debates and use them to train a powerful AI, but then you also want to understand much more than we currently understand about whether that’s actually going to work well. So in some sense, even if there was nothing stopping us from running this kind of training procedure right now, we’re going to have to do a lot of work to understand whether we’re comfortable with that.Do we think that’s good? Or do we think that we should do some other approach or maybe try harder to coordinate to avoid deploying AI. That’s a huge cluster of questions, some of which are empirical questions about how humans think about things. Like what happens in actual debates involving humans, what happens if you actually try and take 20 humans and have them coordinate in the application setting. It also depends on hard philosophical questions. Like I mentioned earlier, the question, “What should a super intelligent AI do?” If you had a formal condition for what it should do, then your problem would be a lot easier.Our current position is we don’t know. In addition to solving that problem, we’re going to be defining that problem. Like should is a tricky word. So that’s the second category of difficulties.There’s a third, big category of difficulties corresponding to … and the third category is maybe something we could just wait on. The current AI is not sophisticated enough to say run interesting debates. That is, if you imagine the kind of debate between humans that’s like interestingly promoting truth, that involves a very complicated learning problem the debaters have to solve. And to think right now, it feels like that problem is just at the limits of our abilities. Like you could imagine in some simple settings training that kind of AI. And so, one option would just be to wait until the AI improves and say we’re going to try and study these techniques in simpler cases and then apply them with the real messiness of human cognition only once the AI’s better.Another option would be to try and push safety out as far as one could go. So it’s actually starting to engage with the messiness of human cognition. And to be clear, the second step I suggested is philosophical difficulties and asking whether this is actually a good scheme. That’s totally going to have to, even today, involve engaging a ton with humans. Like that involves actually running debates, actually doing this kind of decomposition process that underlies amplification.So maybe those are the three main categories of difficulty that I see. I think all of them seem very important. I think my current take is probably that the most important ones are figuring out if this is a good idea rather than being actual obstructions to running the scheme. I think it’s quite realistic to relatively soon be at a place where you could use this procedure to train a powerful AI. And the hard part is just getting to the point where we actually believe that’s a good idea. Or we’ve actually figured out whether that’s a good idea. And then, I mean that’s not just figuring it out, it’s also modifying the procedures so that they actually are a good idea.


Prompt: Yeah that makes a lot more sense now.Okay, let’s push onto a different line of research you’ve been doing into prosaic AI alignment. You’ve got a series of posts about this on ai-alignment.com. Yeah, what’s kind of the argument you’re making? What is prosaic AI?

Response: So, I would describe this as a motivating goal for research, or a statement of what we ought to be trying to do as researchers working on alignment, and roughly what I mean by prosaic AI is AI which doesn’t involve any unknown unknowns, or AI which doesn’t involve any fundamental surprises about the nature of intelligence. So, we could look at existing ML systems and say whether or not I think this is likely, we could ask what would happen if you took these ideas and scaled these ideas up to produce something like sophisticated behavior or human-level intelligence, and then, again, whether or not that’s likely, we can sort of understand what those systems would look like much better than we can understand what other kinds of AI systems would look like just because they would be very analogous to the kinds of systems we could build today.And so, in particular, what that involves, I guess, if the thing we’re scaling up is something like existing techniques in deep learning, that involves defining an objective, defining a really broad class of models, a really giant … and that’s a complicated model involving attention and internal cognitive workspaces, and then just optimizing over that class to find something that scores well according to the objective, and so we’d imagine … yeah, so that’s the class of technique. That’s the basic technique, and you could say, “What would happen if it turned out that that technique could be scaled up to produce powerful AI?” That’s what I mean by prosaic AI, and then the task would be to say, “Supposing you live in that world, supposing we’re able to do that kind of scale-up, can we design techniques which allow us to use that AI for good or allow us to use that AI to do what we actually want, given that we’re assuming that AI can be used to have some really big transformative impact on the world.”Yeah, so there’s a few reasons you might think this is a reasonable goal for research. So, maybe one is that it’s a very … it’s like a concrete model of what AI might look like, and so it’s relatively easy to actually work on instead of sort of being in the dark and having to speculate about what kinds of changes might occur in the field. Second reason is that even if many more techniques are involved in AI, it seems quite likely that doing gradient descent or rich model classes is going to be one of several techniques, and so if you don’t understand how to use that technique safely, it’s pretty likely you’re going to have a hard time.Maybe a third reason is that I think there is actually some prospect that existing techniques will go further than people guess, and that’s a case that’s particularly important from the perspective of alignment, because, in that case, people, sort of by hypothesis, be caught a little bit by surprise. There’s not that much time to do intervening, or to do more work between now and then, so I think in general, I would advocate for a policy of, “Look at the techniques you understand currently and try and understand how to use those techniques to safely use those techniques, and then once you’ve really solved that problem, once you’re like, “Now we understand how to make, how to do gradient descent in a way that produces safe AI,” then you can go on and look towards future techniques that might appear and ideally to understand for each of the techniques that might play a role in building your AI, you’d have some analogous safe version of that technique, which doesn’t introduce problems with alignment but is roughly equally useful.


Prompt: So, I guess the people who wouldn’t be keen on this approach would be those who are confident that current methods are not going to lead to very high levels of general intelligence, and so they expect the techniques that you’re developing now just won’t be usable ’cause they’re gonna be so different.

Response: Yeah, I guess I’d say there’s two categories of people that might be super skeptical of this as a goal. One will be, as you said, people who just don’t believe that existing techniques are going to go that far or don’t believe that they’re going to play an important role in powerful AI systems, and then a second one would be those who think that’s plausible, but that the project is just doomed. That is that there is going to be no way to produce an analog of existing techniques that would be aligned, even if they could in fact play a role in sophisticated AI systems. I think both of those are reasonably common perspectives.


Prompt: Well, I think in a minute we’ll talk about MIRI, and I guess is perhaps a bit of a combination of the two of them.

Response: Yeah, they’re some of the strongest proponents of the second view that we’re super doomed in a world where sophisticated AI looks anything like existing systems.


Prompt: Can you lay out the reasons both for and against thinking that current techniques in machine learning can lead to general intelligence?

Response: Yeah, so I think one argument in favor, or one simple point in favor is that we do believe if you took existing techniques and ran them with enough computing resources, there’s some anthropic weirdness and so on, but we do think that produces general intelligence based on observing humans, which are effectively produced by the same techniques. So, we do think if you had enough compute, that would work. That probably takes, sort of if you were to run a really naïve analogy with the process of evolution, you might think that if you scaled up existing ML experiments by like 20 orders of magnitude or so that then you would certainly get general intelligence.So that’s one. There’s this basic point that probably these techniques would work at large enough scale, so then it just becomes a question about what is that scale? How much compute do you need before you can do something like this to produce human-level intelligence? And so then the arguments in favor become quantitative arguments about why to think various levels are necessary. So, that could be an argument that talks about the efficiency of our techniques compared to the efficiency of evolution, examines ways in which evolution probably uses more compute than we’d need, includes arguments about things like computer hardware, saying how much of those 20 orders of magnitude will we just be able to close by spending more money and building faster computers, which is …20 orders of magnitude sounds like a lot, but actually, you cover … we’ve covered more than 20 orders of magnitude so far, and we will cover a significant fraction of those over the current decade, or you can also try and run arguments on analogies. Look at how effectively or how much compute existing systems take to train and try and understand that. So, you could just try and say, based on what our experience so far, how much compute do you think will be needed?That’s like probably the most important class of arguments in favor. There’s other qualitative arguments like: there are lots of tasks that we’re able to do, so you’d probably want to look at what tasks we have succeeded at or failed at and try and fit those into that quantitative picture to make sense of it. But, I think it’s like not insane to say that existing systems seems like they’ve plausibly reached the level of sophistication of insects, so we are able to take this very brute force approach of doing search over neural nets and get behavior that’s … and this is totally unclear, but I think it’s plausible that existing behaviors as sophisticated as inspects. If you thought that, then I think it would constitute an argument in favor.Yeah, so I guess arguments against, probably the most salient argument against is just, “If we look at the range of tasks humans are able to accomplish, we have some intuitive sense of how quickly machines are able to do more and more of those tasks,” and I think many people would look at that rate of progress and say, “Look, if you were to extrapolate that rate, it’s just gonna take a very, very long time before we’re able to do that many tasks.” I think a lot of this is just people extrapolate things in very different ways. So, some people would look at being able to see the task an insect can do and say, “Wow, insects have reasonably big brains on a scale from nothing to human. We’ve come a substantial fraction of the way. We’re perhaps plausibly going to get there just by scaling this up.”Other people would look at what insects do and say, “Look, insects exhibit almost none of the interesting properties of reasoning. We’ve captured some very tiny fraction of that. Presumably, it’s gonna be a really long time before we’re able to capture even like a small fraction of interesting human cognition.”


Prompt: What are the aspects of cognition that seem most challenging, or, I guess, are most likely to require major research insights rather than just increasing the compute?

Response: Again, with enough compute, you’d sort of expect, or I would be willing to bet that you would get everything in human cognition, and the question is, in some sense, which aspects of cognition are most expensive to produce in this way, or most likely to be prohibitively expensive such that you can’t just find them by brute force search. You have to actually understand them. So, natural things or properties of human cognition operate over very long timescales. Maybe evolution got to take at developing different notions of curiosity until it found a notion of curiosity that is effective, or a notion of play that was effective for getting humans to do useful learning.It’s not clear that you can evaluate. If you have some proposed set of motivations for a human that you’re rolling out, it’s not clear you can evaluate then by actually having a bunch of human lifetimes occur, and so if there’s a thing you’re trying to optimize where every time you have a proposal, you have to check it in order to check it, you have to run a whole bunch of human lifetimes, that’s going to take a whole lot of checks. And so, if there’s like cognitively complicated things that only … right, so maybe curiosity’s simple, but if you have a thing like curiosity that’s actually very complicated or involves lots of moving parts, then it might be very, very hard to find something like that by this brute force search.Things that operate over very short timescales are much, much, more likely to … then you can try a whole bunch of things. You can get feedback about what works, but things that operate over long timescales might be very hard.


Prompt: So, it sounds like you’re saying at some level of compute, you’re pretty confident that current methods would produce human-level intelligence and maybe much more. I think a lot of listeners would find that claim somewhat surprising, or at least being confident that that’s true. Yeah, what’s the reason that you think that?

Response: Yeah, so there’s a bunch of … there are a bunch of things to be said on this topic, so maybe a first thing to say is human intelligence was produced by this process of: try some random genomes, take those genomes which produce the organisms with the highest fitness, and then randomly vary those a little bit and see where you get. In order for that process to produce intelligence, you definitely need a bunch of things. At a minimum, you need to try a huge number of possibilities. Again, now we’re just discussing the claim that with enough compute would work. So, at a minimum, you need to try a whole bunch of possibilities, but you also need an environment in which reproductive fitness is a sufficiently interesting objective.So, one reason that you might be skeptical of this claim is that you might think that the environment that humans evolved in or the lower life evolved in, like, is actually quite complex, and we wouldn’t have access. Even if we had arbitrarily large amounts of compute, it wouldn’t actually be able to create an environment rich enough to produce intelligence in the same way. So, that’s something I’m skeptical of largely because I think … humans operate in this physical environment. Almost all the actual complexity comes from other organisms, so that’s sort of something you get for free if you’re spending all this compute running evolution cause you get to have the agent you’re actually producing interact with itself.I guess, other than that, you have this physical environment, which is very rich. Quantum field theory is very computationally complicated if you want to actually simulate the behavior of materials, but, it’s not an environment that’s optimized in ways that really pull out … human intelligence is not sensitive to the details of the way that materials break. If you just substitute in, if you take like, “Well, materials break when you apply stress,” and you just throw in some random complicated dynamics concerning how materials break, that’s about as good, it seems, as the dynamics from actual chemistry until you get to a point where humans are starting to build technology that depends on those properties. And, by that point, the game is already over. The point when humans are building technologies that really exploit the fact that we live in a universe with this rich and consistent physics, at that point, you already have human-level intelligence. Effectively, there’s not much more evolution occurring.So yeah, maybe on the environment side, I think most of the interesting complexity comes from organisms in the environment, and there’s not much evidence that considerable computational complexity of the world is actually an important part of what gives a human intelligence. A second reason people might be skeptical is they might … this estimate, this 20 orders of magnitude thing would come from thinking about the neurons in all the brains in all the organisms that have lived. You might think that maybe the interesting compute is being done early in the process of development or something about the way that genotypes translate into phenotypes. If you think that, you might think that the neuron counts are a great underestimate for the amount of interesting compute.Or, similarly, you might think other things in the organisms are more interesting than either development or neurons. I think that, like, my main position here is it really does look like we understand the way in which neurons do computing. A lot of the action is sending action potentials over long distances. The brain spends a huge amount of energy on that. It looks like that’s the way that organisms do interesting computing. It looks like they don’t have some other mechanism that does a bunch of interesting computing ’cause otherwise they wouldn’t be spending these huge amounts of energy implementing the mechanism we understand. It does look like brains work the way we think they work.


Prompt: So, I guess some people could think that there’s a lot of computation going on within individual neurons, but you’re skeptical of that.

Response: Yeah, so I think my view would be that mostly the hard thing about … say, if you wanted to simulate a brain, you can imagine there being two kinds of difficulties. One is simulating the local dynamics of neurons, and a second is moving information long distances, say, as you fire action potentials, and I think, most likely, both in the brain and in computers, the movement of information is actually the main difficulty. It’s like, the dynamics within the neuron just don’t … they might be very complicated. It might involve a lot of arithmetic operations to perform that simulation, but I think it’s not hard to compare to just shuffling that around, and shuffling that around, we have a much clearer sense of exactly how much happens because we know that there’s these action potentials. Action potentials communicate information basically only in the timing. I mean, there’s a little bit more than that. But we can basically … we know sort of how much information is actually getting moved.


Prompt: It looks like ones and zeros?

Response: Yeah, it looks like ones and zeros and most of the extra bits are in timing, and we sort of know roughly what level of precision there is, and so there’s not that many bits per action potential.


Prompt: So, I don’t have a lot of understanding of the specifics of how machine learning works, but I would think that one objection people might have is to say that even if you had lots of compute and you tried to make the parameters of this, the machine learning more accurate, just the structure of it might not match what the brain is doing, so it might just cap out at some level of capability because there’s no way for the current methods of … the current way that the data’s being transformed to actually be able to produce general intelligence. Do you think there’s any argument for that or is it just the case that the methods we have now at least at some level of abstraction are analogous to what the human brain is doing, and therefore, with a sufficient amount of compute, maybe a very, very high amount, but they should be able to copy everything that the human brain is doing?

Response: Yeah, so I would say that most of the time, machine learning would be fixed on architecture and then optimize over computations that fit within that architecture. Obviously, when evolution optimizes for humans, it does this very broad search over possible architectures like looking over genomes that encode, “Here’s how you put together a brain.” We can also do a search over architectures, and so the natural question becomes, “How effective are we at searching over architectures compared to evolution?” I feel like this is mostly in the regime of just a computational question. That is, we sort of know … I mean, the very highest level that evolution uses isn’t that complicated, sort of at the meta level, and so you could, in the worst case, just do a search at that same level of abstraction.I guess one point that we haven’t discussed at all but is, I guess, relevant for some, some people would consider super relevant is anthropic considerations concerning the evolution of humans. So, you might think that evolution only extremely rarely produces intelligent life, but that we happen to live on a planet where that process worked.


Prompt: Yeah, what do you make of that?

Response: So, I think it’s kind of hard to make it fit with the evolutionary evidence. This is something that, I think Carl Shulman and Nick Bostrom have a paper about this, and some other people have written about it periodically: I think the rough picture is that intelligence evolves like … if this is the case, if there’s some hard step in evolution, it has to be extremely early in evolutionary history, so in particular, it has to happen considerably before vertebrates, and probably has to have happened by simple worms.


Prompt: And why is that? ‘Cause those steps took longer than the later steps did?

Response: Well, so, one reason … I think the easiest reason to put it before vertebrates is just to say that cephalopods seem pretty smart and the last common ancestor between an octopus and a human is some simple worm. I think that’s probably the strongest evidence. That’s from this paper by Nick and Carl.


Prompt: Okay, because then we have another line that also produced substantial intelligence.

Response: Independently.


Prompt: Independently, and that would be incredibly suspicious if it had happened twice on the same planet, and there, we don’t have the anthropic argument, ’cause you could live on a planet where it only happened once.

Response: That’s right. You could think maybe there’s a hard step between octopi and humans, but then we’re getting into the regime where like sort of any place you look-


Prompt: What is this hard step?

Response: Many things happen twice. Like, birds and mammals independently seem to become very, very intelligent. You could think that maybe in early vertebrates, there was some lucky architectural choice made in the design of vertebrate brains that causes on the entire vertebrate line intelligence will then sort of systematically increase quickly, but what was important was this lucky step, but at some point: you can try and run some argument before you might get stuck before humans. It seems pretty hard to do. It doesn’t seem very convincing and it certainly doesn’t seem like it would give you an argument for why you wouldn’t reach at least like octopus levels of intelligence. So, if you’re thinking that existing techniques are gonna get stuck anywhere around their current level, then this kind of thing isn’t going to be very relevant.


Prompt: Yeah, so I guess it kind of raises a definitional question of, “What is current techniques?” How much do you change the architecture before you say, “Oh, well this is no longer like current machine learning methods. This is no longer prosaic AI?”

Response: Yeah, so I think the thing that’s really relevant from the perspective of alignment research is you want to assume something about what you can do, and the thing you want to assume you can do is: there is some model class. You optimize over that model class given an objective. Maybe you care about whether the objective has to supply gradients. Maybe it doesn’t even matter that much. So then, as an alignment researcher, you say, “Great, the AI researchers have handed us a black box. The black box works as follows. The black box takes some inputs, produces some outputs. You specify how good those outputs were, and then the black box adjusts over time to be better and better.And, as an alignment researcher, as long as something fits within that framework, you don’t necessarily care about the details of, “What kind of architecture are we searching over? Are you doing architecture search? Or what form does the objective take?” Well, what form the objective takes, you may care about, but most other details you don’t really care about because alignment research isn’t going to be sensitive to those details.So, in some sense, you could easily end up with a system: existing ML researchers would say, “Wow, that was actually quite a lot different from what we were doing in 2018,” but which an alignment researcher would say “That’s fine. The important thing from my perspective is this still matches with the kind of alignment techniques that we were developing,” so we don’t really care how different it looks. We just care about, did it basically change the nature of the game from the perspective of alignment?


Prompt: Yeah. Can we look backwards in history and say, would techniques that we developed five or ten years ago work on today’s architectures?

Response: Yeah, so we can look back. Hindsight is always complicated and hazardous, but I think you would say, if you were to, say, in 1990, perform a similar exercise and look across techniques, I would say certainly the kinds of things we’re talking about now would exist. That would be part of your picture. They would not have nearly as much, be nearly as much of a focal point as they are today because they hadn’t yet worked nearly as well as they worked now, so I guess we would be talking about what fraction of your field of view would these techniques occupy?So, I think it’s pretty safe to say that more than 10% of your field of view would have been taken up by the kind of thing we’re discussing now, and the techniques developed with respect, with that 10% of possibilities in mind would still apply. Existing systems are very, very similar to the kinds of things people are imagining in the late ’80s, and there’s a question like, “Is that number 10% or is it a third?” I think that’s pretty unclear and I don’t have enough of a detailed understanding of that history to really be able to comment intelligently, and I’d wanna defer to people who were doing research in the area at that time.I do think the … if you had instead focused on different kinds of techniques, like if you’d been around in the further past and you were, say, trying to do AI alignment for expert systems, I don’t feel that bad about that. I guess some people look back on history and say, “Man, that would have been a real bummer if you’d been alive in the ’60s and you’d done all this AI alignment research that didn’t apply to the kind of AI we’re building now,” and my perspective is kind of like, “Look, one, if it takes 50 years to build AI, it doesn’t matter as much what the details are of the AI alignment work you did in the ’60s. Two, actually, there’s a lot of overlap between those problems like many of the philosophical difficulties you run into alignment are basically the same, even between existing systems and expert systems.Three, I would actually be pretty happy with the world where like when people propose a technique, a bunch of AI alignment researchers invest a bunch of time understanding alignment for expert systems, and then 15 years later, they move onto the next thing. It’s like not that bad a world. I expect you would, in fact … if you just solved this sequence of concrete problems, that actually sounds pretty good. It sounds like a good way to get practice as a field. It sounds reasonably likely to be useful. There’s probably lots of commonalities between those problems. Even if they turn out to be totally wasted, it’s a reasonable bet an expectation: you sort of have to do … that’s the cost we have to pay if you want to have done a bunch of research for the techniques that are actually relevant unless you’re very confident the current techniques are not the things that will go all the way, or that it’s doomed. I think both those positions seem really, really hard to run to me. I haven’t heard very convincing arguments for either of those positions.


Prompt: What’s expert systems?

Response: The systems based on having a giant set, maybe reasoning rules and facts, and then they use these rules to combine these facts.


Prompt: And that just didn’t work?

Response: Yeah, there was a period where people were more optimistic about them. I don’t know the history very well at all. I think in general, certainly, it didn’t realize the ambitions of the most ambitious people in that field, and certainly, it’s not the shape of most existing, or the kinds of systems people are most excited about today.


Prompt: Okay, so, we mentioned that a group that has kind of a different view from this prosaic AI is the Machine Intelligence Research Institute at Berkeley. If I understand correctly, you got into AI safety in part through at least that social group or that intellectual group, but it seems like now, you kind of recommend … you kind of represent a different node or, what’s the term, access within the people working on AI safety. Yeah, how would you describe their view and how it differs from yours today?

Response: I would say the most important difference is they believe this prosaic AI alignment project is very likely to be doomed. That is, they think if the shape of sophisticated AI systems resembles the shape of existing ML systems, or if it in particular: you obtain sophisticated AI by defining a model class, defining an objective and doing gradient descent, finding the model that scores well according to the objective, then they think we’re just extremely doomed such that they think the right strategy is to instead step back from that assumption and say, “Can we understand other ways that people could build sophisticated AI?”Part of that is like, if you’re doing gradient descent or this big model … if you’re doing gradient descent to find a model that performs well, you’re gonna end up with some actual particular models. You’re gonna end up with some particular way of thinking that your giant neural net embodies, and you could instead … instead of just specifying procedures that give rise to that way of thinking, you could actually try and understand that way of thinking directly and say, “Great, now that we understand this, we can both reason about its alignment and maybe we can also design it more efficiently or we can more efficiently describe search procedures that will uncover it once we know what it is that they’re looking for.”And, I’d say that’s like the biggest difference, and the crux there is mostly the: is it possible to design alignment techniques that you make something like existing ML systems safe? And so, my view is that, mostly likely, that’s possible. Not most, like, more likely than not, not like radically more likely than not, but somewhat more likely than not, that’s possible and that as long as it looks possible and you have attractive lines of research to pursue and a clear path forward, we should probably work on that by default, and that we should … at some point, if it’s like, “Wow, it’s really hard to solve alignment for systems that look anything like existing ML,” then you really wanna understand as much as you can why that’s hard, and then you wanna step back and say, “Look, people in ML, it looks like the thing you’re doing actually is sort of like unfixably dangerous and maybe it’s time for us to think about weird solutions where we actually have to change the overall trajectory of the field based on this consideration about alignment.”If it’s not reasonable to call that weird, from the outside view, you might think, “Well, the goal of AI is to make things good for humans. It’s not crazy to change the direction of the field based on what is plausibly going to be alignable.”


Prompt: But it would seem strange to them today?

Response: Yeah, people in ML would not be like, “Oh, that makes a lot of sense. Let’s just swap what we’re doing.” So, I guess, my position would be like, “I’m currently much, much more optimistic than MIRI people.” I think that’s the main disagreement about whether it will be possible to solve alignment for prosaic AI systems, and I think as long as we’re optimistic in that way, we should work on that problem until we discover why it’s hard or solve it.


Prompt: Just to make it more concrete for people, what are the kind of specific questions that MIRI is researching that they think are useful?

Response: I think, at this point, MIRI’s public research, the stuff that they publish on and talk about on the internet, one big research area is decision theory, so understanding, supposing that you have some agent which is able to make predictions about things or has views on the truth of various statements, how do you actually translate those views into decision? This is tricky, ’cause you want to say things like, “You care about quantities. What would happen if I were to do X,” and it’s not actually clear what would happen if I were to do X means. Is it causal, kind of factual, like a certain kind of statement? It’s not clear that’s the right kind of statement.


Prompt: So this is causal decision theory, evidential decision theory and-

Response: Yeah, and most of the stuff they’re considering seriously is like … like once you get really precise about and you’re like, “We’d like this to be an algorithm,” the whole picture just gets a lot weirder and there are a lot of distinctions that people don’t normally consider in the philosophy, philosophical community, that are kind of, “You have to wade through if you want to have … be in the place where you have a serious proposal for,” here is an algorithm for making decisions given as input like views on these empirical questions or given as input like a logical inductor or something like that.So, that’s one class of questions that they work on. I think another big class of questions they work on is like … I mean like stepping back from, looking at the whole problem and saying from a conceptual perspective, supposing you grant this view, this worldview about what’s needed, what … I don’t know a good way to define this problem. It’s kind of just like, “Figure out how you would build an aligned AI,” which is a good problem. It’s the very high level problem. I endorse some people thinking about the very high level problem. I think it’s one of the more useful things to think about. There’s some flavor of it that depends on what facts you … what you think are the important considerations or what you think the difficulties are, so they work on a certain version of that problem.I think other examples of things include … they’re very interested just in, “What are good models for rational agency?” So, we have such models in some settings like in Cartesian settings where you have an environmented agent that communicate over some channel where they send bits to one another. It becomes much less clear what agency means once you have an agent that’s physically substantiated in some environment. That is, what does it mean to say that a human is like a consequentialist thing acting in the world, given that the human is just actually like: some of the degrees of freedom in the world are leading together in this complicated way to make a human. It’s quite complicated to talk about what that actually means, that there’s this consequentialist in the world. That’s a thing they’re like super interested in.Yeah, figuring out how to reason about systems that are logically very complex including systems that contain yourself or contain other agents like you. How do we formulize such reasoning, is another big issue.


Prompt: Does MIRI have a different view as well about the likelihood of current methods producing general intelligence or is that-

Response: There’s probably some difference there. There’s a lot less stark than the other one. I think maybe a difference in that space that’s more close is like: I kind of have a view that’s more like: there’s probably some best way. There’s some easiest way for our current society as it’s currently built to develop AI, and the more of a change you want to make from that default path, the more difficult it becomes. Whereas I think the MIRI perspective would more be like: the current way that we build ML is reasonably likely to just be very inefficient, so it’s reasonably likely that if you were to step back from that paradigm and try something very different, that it would be comparably efficient and maybe more efficient, and I think that’s a little bit … yeah, I guess I don’t buy that claim. I don’t think it’s as important as the definite doom claim.


Prompt: So, what are the best arguments for MIRI’s point of view that the current methods can’t be made safe?

Response: So, I’d guess I’d say there’s two classes of problems that they think might be unresolvable. One is: if I perform … if I have some objective in mind … suppose I even have the right objective, an objective that perfectly tracks how good a model is, all things considered according to human values, and then I optimize really aggressively on that objective. The objective is still just a feature of the behavior of this: I have this black box I’m optimizing over the weights of some neural network, so now I have an objective that perfectly captures whether the behavior is good for humans, and I optimize really hard on that objective.So one of MIRIs big concerns is that even if we assume that problem is resolved and you have such an objective, then it’s pretty likely you’re going to find a model which only has these desirable properties like on the actual distribution where it was trained, and that it’s reasonably likely that in fact, that system that you’ve trained is going to be some consequentialist who wants something different from human flourishing and just happens on the trained distribution to do things that look good.


Prompt: Within that narrow range?

Response: Yes. So, an example of this phenomenon I that think MIRI people think is pretty informative, though certainly not decisive on its own, is like: humans were evolved to produce lots of human offspring, and that it’s the case that humans are sophisticated consequentialist whose terminal goal is not just producing offspring, so that even though the cognitive pulse that humans use is very good for producing offspring over human evolutionary history, it seems like it’s not actually great, sort of already broken down to a considerable extent, in the long run looks like it will break down to a much, much greater extent.So, then, if you were like a designer of humans being like, “I know. I’ve defined this objective that tracks how many offspring they have. Now I’m going to optimize it over many generations. I’m gonna optimize biological life over a million generations to find the life which is best at producing offspring,” you’d be really bummed by the results. So, their sort of expectation is that in a similar way, we’re going to be really bummed. We’re going to optimize this neural net over a very large number of iterations to find something that appears to produce actions that are good by human lights, or we’re going to find something whose relationship to human flourishing is similar to like humans’ relationship to reproduction where they sort of do it as a weird byproduct of a complicated mix of drives rather than because that’s the thing they actually want, and so when generalized, they might behave very strangely.


Prompt: Okay, sounds kind of persuasive. What’s the counterargument?

Response: So, I think there’s a few things to say in response. So, one is that evolution does a very simple thing where you sample environments according to this distribution, then you see what agents perform well on those environments. We, when we train ML systems, can be a little bit more mindful than that, so in particular, we are free to sample from whatever distribution over environments we are … any distribution we’re able to construct, and so as someone trying to solve prosaic AI alignment, you are free to look at the world and say, “Great, I have this concern about whether the system I’m training is going to be robust, or whether it might generalize in a catastrophic way in some new kind of context,” and then I’m free to use that concern to inform the training process I use, so I can say, “Great, I’m going to adjust my training process by, say, introducing adversary and having the adversary try and construct inputs on which the system is going to behave badly.”That’s something that people do in ML. it’s called adversarial training, and if you do that, that’s very different from the process evolution ran. Right now, you imagine that there’s someone roughly as smart as humans who’s like, constructing these weird environments, like if they’re looking at humans and say, “Great, the humans seem to care about this art shit,” then the adversary’s just constructing an environment where humans have lots of opportunities to do art or whatever, and then if they don’t have any kids, then they get down-weighted.If there’s some gap, if there’s some context under which humans fail to maximize reproductive fitness, then adversary can specifically construct those contexts, and use that to select against. Again, the reproductive fitness analogy makes this sound kind of evil, but you should replace reproductive fitness with things that are good.Yeah, so that’s one thing. I think the biggest thing, probably, is that, as the designers of the system, we’re free to do whatever we can think of to try and improve robustness, and we will not just like sample-


Prompt: Yeah, we can look forward rather than just look at the present generation.

Response: Yeah, although it’s a challenging problem to do so, so that’s a thing a bunch of people work on. It’s not obviously they’ll be able to succeed, certainly. I don’t think that like this analogy should make you think, like … I think the analogy maybe says there is a problem. There is a possible problem, but doesn’t say, “And that problem will be resistant to any attempt to solve it.” It’s not like evolution made a serious attempt to solve the problem.


Prompt: Yeah. If you can make the method courageables that you can continue improving it, changing it, even as you’re going with an AI transforming in the wild, that seems like it would partially solve the problem, ’cause one of the issues here is that humans ended up with the motivations that they have, desires that they have, and then we’re going about it in a single generation or a handful of generations in evolutionary time, changing everything about the environment. The environment’s changing much faster than we are such that we’ve become … our drives no longer match what would actually be required to reproduce at the maximal rate, whereas if you were changing humans as we went, as our behavior ceased to be adaptive from that point of view, then perhaps you could keep us in line so we’d be fairly close to the maximal reproductive rate. Does that make sense?

Response: Yeah. I think that’s like an important part of the picture for why we have hope. That is, if you’re like, “Yeah, we’re gonna evolve a thing that just wants human flourishing, or we’re gonna like, do grand ascent until we find a thing that really wants me to flourish and then we’re gonna let it rip in the universe,” that doesn’t sound great. But, if you’re like, “We’re gonna try and find a thing which helps humans in their efforts to continue to create systems that help humans continue to create systems that help humans achieve, help humans flourish,” then that’s … I guess you could imagine in the analogy: instead of trying to evolve a creature which just cares about human flourishing, you’re trying to evolve a creature that’s really helpful, and somehow, “Be really helpful and don’t kill everyone and so on,” is like an easier, a more imaginable set of properties to have, sort of even across a broad range of environments than matching some exact notion of what constitutes flourishing by human lights.I think one reason that people at MIRI are from a similar school of thought would be pessimistic about that is they have this mental image of humans participating in that continuing training process, sort of training more and more sophisticated AIs. If you imagine a human is intervening and saying, “Here’s how I’m gonna adjust this training process,” or, “Here’s how I’m going to shape the course of this process,” it sounds kind of hopeless, because humans are so much slower, and in many respects, presumably so much less informed, less intelligent.


Prompt: That they might just be adding noise?

Response: Yeah. And it would be very expensive to have human involvement. Mostly, they wouldn’t be presuming to give direction, some random direction. I think the main response there is like: you should imagine humans performing this process … early on in this process, you should imagine humans being the ones adjusting objectives or adjusting the behavior of the system. Later on, you should imagine that as mostly being carried out by the current generation of AI systems, so the reason that humans can keep up is that process goes faster and faster, and so hopefully because we’re maintaining this property that there always a whole bunch of AI systems trying to help us get what we want.


Prompt: Will it continue to bottom out in some sense, what humans say about how the upper level is going? I’m imagining if there’s multiple levels of the most advanced AI then the less advanced, I guess this is kind of what we were talking about earlier. Then you’ve got kind of humans at the bottom. At some point, would they just disappear from the equation and it’s like all-

Response: Yeah, so it’s always going to be anchored to what a human would have said. In some sense, that’s like the only source of ground truth in the system. Humans might not actually be … there might be some year beyond which humans never participate, but at that point, the reason that would happen would be because there is some system. Suppose in the year 2042, humans stop ever providing any input to AI systems again, the reason that would be possible is that in the year 2042 there was some AI system which we already trust robustly, “Do well enough according to human lights.”


Prompt: And it can do it faster and cheaper.

Response: Yeah. It’s a little bit tricky to ever have that handoff occur, because that system in 2042, the one that you’re trusting to hand things off to, has never been trained on things happening in 2043, so it’s a little bit complicated, and it’s not that you’re gonna keep running that same system in 2042. It’s that that system is going to be sufficiently robust that it can help you train the system in 2043 that it’s going to … yeah.


Prompt: Yeah, if you could visit 50 years in the future and see everything that was happening there, how likely do you think it would be that you would say, “My view of this was broadly correct,” vs, “MIRI’s view was more correct than mine with hindsight?” I’m trying to measure how confident you are about your general perspective.

Response: Yeah, so I certainly think there are a lot of cases where it would be like, “Well, both views were very wrong in important ways,” and then you could easily imagine both sides being like, “Yeah, but my view was right in the important way,” so that’s certainly a thing which seems reasonably likely. In terms of thinking in retrospect that my view was like unambiguously right, I don’t know, maybe I’m on like, relative to MIRI’s view, maybe I’m at 50-70% … that’s pretty high, whatever, like 50-60% that like in retrospect we’ll be like, “Oh, yeah, this was super clear,” and then maybe on the other side I would put a relatively small probability that’s super clear in the other direction, that like maybe 20% or 10% or something that like in retrospect I’m like, “Geez, I was really wrong there. Clearly my presence in this debate just made things worse.”


Prompt: And then there’s kind of a middle ground of both of them had important things to say, yeah?

Response: Yeah.


Prompt: Interesting. So, you’re like reasonably, reasonably confident, but it seems like but you would still support, given those probabilities, MIRI doing substantial research into their line of inquiry.

Response: Yeah, I’m excited about MIRI doing the stuff MIRI’s doing. I would prefer that MIRI people do things that were better on my perspective, which I suspect is most likely to happen if they came to agree more with this perspective.