Prompt: But at some point, let’s say that your line of research had four times as many resources or four times as many people, then you might say, “Well, having one more person on this other thing could be more useful,” even given your views, right?

Response: Yeah, although, I don’t think … the situation is not like there’s line of research A and line of research B and the chief disagreement is about which line of research to pursue. It’s more like: if I was doing something very, very similar to what MIRI’s doing, or doing something superficially quite similar, I would do it in a somewhat different way, so, to the extent I was working on philosophical problems that clarify our understanding of cognition or agency, I would not be working on the same set of problems that MIRI people are working on, and I think those differences matter probably more than the high-level, “What does the research look like?” So, lots of stuff in the general space of research MIRI’s doing that I’d be like, “Yep, that’s a good thing to do,” which, now we’re in this regime of, yeah, it depends on how many people are doing one thing vs doing the other thing.


Prompt: Do you think that those … if your view is correct, is there going to be much like incidental value from the research that MIRI is doing or is it kind of just by the by at that point?

Response: So, one way research of the kind MIRI’s doing is relevant is to clarifying whether, when they talk about amplification or debate, they each have this conceptual, key conceptual uncertainty. In the case of debate, is it the case that debates lead to … that the honest strategy, telling the truth, saying useful things is actually a winning strategy in a debate, or in the case of amplification, is there some way to assemble some large team of aligned agents such that the resulting system is smarter than the original agents and remains aligned. Those conceptual difficulties seem not at all unrelated to the kinds of conceptual … like if you’re asking, “How would we build an aligned AI using infinite amounts of computing power without thinking at all about contemporary ML?” That’s a very similar kind of question, thinking, “What are the correct normative standards of reasoning that you should use to evaluate computing claims, what are, when you compose these agents, like, what kind of decomposition of cognitive work is like actually alignment preserving or do we expect to produce correct results?So, a natural way in which the kind of research MIRI’s doing could add value is by shedding light on those questions. And an expectation, I’d guess they’re at least several times less effective at answering those questions than if they were pointed at them more directly, you know? I don’t know if it’s like five times less effective than if they were pointed at them directly. I think it’s a smaller multiple than that, probably.


Prompt: What would you say to people who are listening who just feel kind of agnostic on whether prosaic AI approach is the best or MIRI’s is?

Response: You mean what would I say in terms of what they ought to do?


Prompt: Yeah, or maybe what they ought to think or things to consider if they maybe don’t feel qualified to judge this debate.

Response: Sure, in terms of what to do, I suspect comparative advantage considerations will generally loom large, and so if one’s feeling agnostic, those will likely end up dominating or, comparative advantage plus short-term what would be the most informative, involve the most learning, build the most flexible capital. In terms of what to think, all things considered, I don’t know. That seems pretty complicated. It’s going to depend a lot on what kind of expertise, just in general, looking at a situation with conflicting people who’ve thought a lot about the situation, how do you decide whose view to take seriously in those cases? To be clear, the spectrum of views amongst all people, my view is radically closer to MIRI’s than almost anyone else in the machine learning community on most respects. There are other respects in which the machine learning community is closer to MIRI than I am. So like, the actual menu of available views is unfortunately even broader than this one.


Prompt: If I’m broader than Paul Christiano’s view and MIRI’s generalized view.

Response: Indeed.


Prompt: It’s unfortunate. You’re saying there’s a third option?

Response: Yeah. In fact, yeah, it’s quite a lot broader. Like, I think being agnostic is not a crazy response. I think there’s an easy position where, well, the most confident claims, sort of all these perspectives like differ substantially on emphasis, but one could basically could significant probability on all of the most confident claims from every perspective. Yeah, certainly the convex combination of them will be more accurate than any particular perspective, and then in order to do significantly better than that, you’re going to have to start making more seriously claims about who you’re willing to ignore.


Prompt: Who to trust, yeah. What would you say is kind of the third most plausible broad view?

Response: I think one reasonably typical view in the machine learning to which I’m sympathetic is the, “All of this will be mostly okay.” As AI progresses, we’ll get a bunch of empirical experience messing around with ML systems. Sometimes they’ll do bad things. Correcting that problem will not involve heroic acts of understanding-


Prompt: Safety specifically, or alignment, specifically, not beyond what might happen anyway.

Response: Yeah, and it’s a little bit hard. You could separate that out into both the claim about what will happen anyway and a claim about what is required. I guess, the views we were talking about for me and MIRI were more about what is required. We’ve separate disagreements about what will likely happen. I think there’s a different ML position on what is likely to be required, which says more like, “Yeah, we have no idea what’s likely to be required. It’s reasonably likely to be easy, any particular thing we think is reasonably likely to be wrong,” and that’s like … I could try and flesh out the view more, but roughly, it’s just like, “We don’t know what’s going to happen and it’s reasonably likely to be easy, or by default, expected to be easy.”I think there’s a reasonable chance in retrospect that looks like a fine view. Yeah, I don’t see how to end up with high confidence in that view, and if you’re like a 50% chance of that view, it’s not gonna have that huge an effect.


Prompt: On your expected value of working on safety, yeah?

Response: Yeah, it may-


Prompt: It only halves it at worst, yeah? Or at best.

Response: Yeah, and increasing that probability from … if you give that a significant probability, that might matter a lot if you have a, “We’re definitely doomed,” view. So, I think on the MIRI view, maybe accepting, giving significant credence to the machine learning perspective would significantly change what they would do, ’cause they currently have this view where you’re kind of at like zero. I don’t know if you’ve seen this post on the logistics success curve that Eliezer wrote.


Prompt: I haven’t.

Response: The idea is that if you’re close to zero, then most interventions … if your probability of success is close to zero, then most interventions that look common-sensically useful aren’t actually going to help you very much ’cause it’s just going to move you from like 0.01% to 0.02%.


Prompt: So this would be a view that’s kind of, “You need many things all at once to have any significant chance of success,” and so just getting one out of 100 things you need doesn’t move you much.

Response: That’s right, just making organizations like a little bit more sane or fixing one random problem here or one random problem there isn’t much going to help. So, if you have that kind of view, it’s kind of important then, you’re putting really low probability on this. This isn’t the only perspective in ML, but it’s one conventional ML perspective. I think on my view, it doesn’t matter that much if you give that 30% or 50% or 20% probability. I think that probability’s not small enough that you should discount that case, interventions that are good, if the problem’s not that hard, it seems like they’re likely to be useful, and also it’s not high enough that-


Prompt: I would have thought that it would make little difference to your strategy ’cause in that case things would be, “Okay, you don’t really have to do anything,” so you can almost just … even if you think it’s 50/50 whether any of this is necessary or not, you can just largely ignore that.

Response: Yeah, that’s what I’m saying. It doesn’t make a huge difference. A way in which it matters is like: you might imagine there are some interventions that are good, like in worlds where things are hard, and there’s 50/50 and those interventions are still good. Maybe they’re half as good as they otherwise would have been, like we were saying, and there’s some interventions that are good in worlds where things are easy. That is, you might be like, “Well, if things were easy, we could still fuck up in various other ways and make the world bad,” and so, reducing those probabilities, I would say that’s also a valuable intervention because the probability of things-are-easy is not low enough that that’s getting driven down to zero.


Prompt: So then just more normal world improvement, or making it more likely that we encode good values? So, if the alignment problem is solved, then it becomes more a question of, “What values will we in fact program into an AI?” And trying to make sure that those are good ones.

Response: Yeah, there’s a lot of things that actually could come up in that world. So, for example, your AI could have a very … if an AI has a very uneven profile of abilities. You could imagine having AI systems that are very good at building better explosives or designing more clever biological weapons that aren’t that good at helping us accelerate the process of reaching agreements that control the use of destructive weapons or better steering the future.So, another problem independent of alignment is this uneven abilities of AI problem. That’s one example. Or we might just be concerned that as the world becomes more sophisticated, there will be more opportunities for everyone to blow ourselves up. We might be concerned that we will solve the alignment problem when we build AI, then someday that AI will build future AI and it will fill out the alignment problem. So, there’s lots of extra problems you could care about.


Prompt: I suppose there’s also: AI could be destabilizing to international relations or politics or be used for bad purposes even though if … so we can give it good instructions and we’ll give it instructions to cause harm.

Response: Yeah, so then there’s a question of how much you care about that kind of destabilization. I think most people would say they care at least some. Even if you have a very focused-on-the-far-future perspective, there’s some way in which that kind of destabilization can lead to irreversible damage. So yeah, there’s a bunch of random stuff that can go wrong with AI and you might become more interested in attending to that or saying, “How do we solve those problems with a mediocre understanding of alignment if a mediocre understanding of alignment doesn’t automatically doom you?”


Prompt: Yeah, is there anything else you wanna say on, I guess, MIRI before we move on? Obviously, at some point, I’ll get someone on from there to defend their view and explain what research they think is most valuable, hopefully some time in the next couple of months.

Response: Yeah, so I guess one thing I mentioned earlier, there were like these two kinds of concerns or two kinds of arguments that would give that we’re super doomed on prosaic if AI looks like existing ML systems. I mentioned one of them, this, even if you have the right objective, it’s plausible that the thing you produce will have some other real objective of the consequentialist who just incidentally is pursuing that objective on a training distribution.There’s a second concern that actually constructing a good objective is incredibly difficult, so in the context of the kinds of proposals that I’ve been discussing, like in context of iterative amplification, they’d then be saying, “Well, all the magic occurs in the step where you aggregate a bunch of people to make better decisions than those people who made it alone,” and in some sense, any way that you try and solve prosaic AI alignment is going to have some step like that where you are implicitly encoding some answer to the alignment problem in the limit of infinite computation.‘Cause they might think that that problem like alignment in that limit is still like sufficiently difficult or has all the core difficulties in it so that it’s not clear. This might say that we’re doomed under prosaic AI alignment, but more directly, it would just say, “Great, we need to solve that problem first, anyway,” ’cause there’s no reason to work on the distinctive parts of prosaic AI alignment rather than trying to attack that conceptual problem and then learning that we’re doomed or having a solution which we could then … maybe it would give you a more direct angle of attack.


Prompt: So, you’re on the board of a newish project called Ought. What is that all about?

Response: So, the basic mandate is understanding how we can use machine learning to help humans make better decisions. The basic motivation is that we are super interested: if machine learning makes the world a lot more complicated and is able to transform the world, we want to also ensure that machine learning is able to help humans understand those impacts and steer the world in a good direction. That’s, in some sense, what the alignment problem is about. You want to avoid a situation that’s gonna mismatch between how well AI can help you develop new technologies and how well AI can help you actually manage this more complicated world it’s creating.I think the main project, or certainly, that project that I am most interested in is on what they call factored cognition, which is basically understanding how you can take complex tasks and break them down into pieces where each piece is simpler than the whole, so it doesn’t depend on the whole context, and then compose those contributions back to solve the original task.So, you could imagine that in the context of taking a hard problem and breaking down to pieces that individual humans can work on, so like, say, a hundred humans, you don’t want any of them to have to understand the entire task. You want to break off some little piece that that person can solve, or you can think of it in the context of machine learning systems. In some sense, the human version is most interesting because it is a warmup or a way of studying in advance the ML’s version.So, in the ML version, that would be, now instead of 100 people, you have some people and a bunch of ML systems, which have some set … maybe an ML system has more limited ability to respond to complex context, like a human has a context in mind when they articulate this problem. ML system has some limited ability to respond to that context, or fundamentally, I think the most interesting reason to care about breaking tasks down to small pieces is because once you make the task simpler, once an ML system is solving some piece, it becomes easier to evaluate its behavior and whether it’s behavior is good.So, this is very related to in the way that iterated amplification hopes to solve AI alignment is by saying we can inductively train more and more complex agents by composing weaker agents to make stronger agents, so this factor cognition project is … it is one possible approach for composing a bunch of weaker agents to make a stronger agent, and in that sense, it’s like one of the main … addressing one of the main ingredients you would need for iterated amplification to work.I think right now it’s kind of the main project that’s aiming at acquiring evidence about how well that kind of composition works, again, in the context of like just doing it with humans since humans are something we can study today. We can just recruit a whole bunch of humans. There’s like a ton of work in like actually starting to resolve that uncertainty, and we can learn about … there’s a lot of work we’ll have to do before we can be able to tell, “Does this work? Does this not work?” But I’d say that’s one of the main things I was doing right now, the reason I’m most excited about it.


Prompt: Is this a business?

Response: It’s organized as a nonprofit.


Prompt: Is Ought hiring at the moment and what kind of people are you looking for?

Response: Yeah, so I think there are some roles that will hopefully be resolved by the time we’re hired for, by the time this podcast comes out. Some things that are likely to be continuing hires are researchers who are interested in understanding this question, understanding and thinking about how you compose the small contributions and solutions to harder tasks, and that’s a … there are several different disciplines that potentially bear on that, but sort of people who are interested in computer science, are interested in like … the approach they’re taking, things that are interested in programming languages are also a reasonable fit, people who are just … I think there’s some stuff that doesn’t fit will including in the academic discipline but if you just think about the problem, “How do you put together a bunch of people? How do you set up these experiments? How do you help humans be able to function as parts of a machine?”So, researchers who are interested in those problems is one genre, and another is engineers who are interested in helping actually build systems that will be used to test possible proposals or will substantiate the best guess about how to solve those proposals, and those will be … in contrast, Open AI is hiring researchers and engineers in ML, so sort of engineers would then be building ML systems, testing ML systems, debugging and improving and so on, ML systems. I think at Ought, similarly hiring both researchers and engineers and people in between, that they are the focus that’s lost on ML. It’s more on, again, building systems that will allow humans to … humans and other simple automation to collaborate, to solve hard problems, and so it is more … it involves less of a distinctive ML background. It’s more potentially a good fit for people who have software engineering background and the problem’s interesting and they have some relevant background, or just the problem’s interesting and they have a broad background in software engineering.


Prompt: Okay, well, I’ll stick up a link to that, to the Ought website with more information on specifically what it’s doing and I guess what vacancies are available whenever we manage to edit this and get it out.

Response: Cool.


Prompt: Okay let’s talk about what listeners who are interested working on this problem could actually do and what advice you have for them. So we’ve had a number of episodes on AI safety issues, which have covered these topics before with Dario Amodei, your colleague, as I mentioned. Jan Leike at DeepMind. As well as Miles Brundage and Allan Dafoe at FHI working on more policy and strategy issues. Do you have a sense of where your advice might deviate from those of those four people or just other people in general on this topic?

Response: So I think there’s a bunch of categories of work that need to be done or that we’d like to be done. I think I’d probably agree with all the people you just listed about. Each of them, presumably, would have advocated for some kind of work. So I guess Dario and Jan probably were advocating for machine learning work that really tried to apply or connect ideas about safety to our actual implementations. Filling up the engineering expertise to make these things work. And acquiring empirical evidence about what works and what doesn’t. And I think that project is extremely important. And I’m really excited about EA’s training up in ML and being prepared to help contribute to that project. Like figuring out whether ML’s a good fit for them. And then, if so, contributing to that project. I guess I won’t talk more about that because I assume it’s been covered on previous podcasts.I’d probably also agree with Miles and Allan about there’s like a bunch of policy work and strategic work that seems also incredibly important. I also won’t talk more about that.I think some categories of work that I consider important that I wouldn’t expect those people to mention. I think for people who with a background in computer science, but not machine learning. Or who don’t want to work in machine learning have decided that’s not the best thing. Don’t enjoy machine learning. I think there’s a bunch of other computer science work that’s relevant to understanding the mechanics and proposals. Like debate or amplification.So an example would be like, right now, Ought, one of their projects is on factored cognition. So, in general, on how you can take a big task and decompose it into pieces which don’t depend on the entire context, and then put those pieces together in a way that preserves the semantics of the individual agents or the alignment of the individual workers. So that’s a problem which is extra important in the context of machine learning or in the context of endured amplification. But that one can study almost entirely independent of machine learning.It is one can just say, like let’s understand the dynamics of such decomposition. Let’s understand what happens when we apply simple automation to that process. Let’s understand what tasks we can decompose and can’t. Let’s understand what kind of interface or what kind of collaboration amongst agents actually works effectively. So that’s an example of a class of questions which depend on a sort of well studied from a computer science perspective, but aren’t necessarily machine learning questions. Which I’d be really excited to see work on. And there’s similar questions in the debate space where just understanding how do we structure such debates. Do the lead to truth? Etc.I think one could also study those questions not from a computer science perspective at all. But I think it’s like super reasona … Like, I don’t know. I think philosophers differ a lot in their taste. But like, for example, if you’re a philosopher interested in asking a question about this area, then I think under what conditions do debate lead to truth is not really a question about computers in any sense. It’s the kind of question that falls under computer scientist’s sensibilities, but I think that taking a really technical but not necessarily quantitative approach to that question is accessible to lots of people who want to try and help with AI safety. And similarly for amplification.So I think in both of those areas, there’s questions that could be studied from a very computer science perspective and involve software engineering and involve running experiments. And is also can be studied from a more philosophical perspective. Just thinking about the questions and about what we really want and alignment works.They can also be studied from this more psychology perspective of actually engaging. Like some of them are going to run relatively large scale experiments involving humans. I don’t know if things are … like if the time is right for that. But that’s definitely, there’s definitely experiments in the space that do seem valuable. And it seems like that at some point in the future there’s going to be more of them.


Prompt: Sorry, what do you mean by that?

Response: So if you ask how does this kind of decomposition work or how do these kinds of debates work? Like the decomposition is ultimately guided by … Right, so I originally described this process involving a human and a few AI assistants. Ultimately you want to replace that human with an AI that’s predicting what a human would do. But nevertheless, the way that you’re going to train that system or the way we currently anticipate training that system involves a ton of interaction. Like I’m assuming there’s really just imitating or maximizing the approval of some human who’s running that process. And so, in addition to caring about how machines work you care a ton about how does that process work with actual humans? And how can you collect enough data from humans to … How can you cheaply collect enough data from humans that you can actually integrate this into the training process of powerful AI systems.So I don’t think that’s a fact about … That doesn’t bear on many of the traditional question psychology and maybe that’s a bad thing to refer to it as. But it is like a … It involves studying humans. It involves questions about particular humans and about how humans behave. About how to effectively or cheaply get data from humans. Which are not really … They’re questions machine leaner people have to deal with because we also have to deal with humans. But really it’s like a much larger … Machine leaner people are not that good at dealing with the interaction with humans at the moment.So yeah. So that’s some family of questions. I think the ones I’m most excited about are probably more on the philosophical computer science bent. There are lots of people who wouldn’t be a great fit for working in the ML. Who wouldn’t be great for working on those questions.I think also stepping back further, setting aside the amplification of debate, I think there’s just still a lot of very big picture questions about how do you make AI safe? That is you could focus on some particular proposal, but you could also just consider the process of generating additional proposals or understanding the landscape of possibilities, understanding the nature of the problem. I don’t know if you’ve ever had anyone from MIRI on, but I’m sure they would advocate for this kind of work. And I think that’s also … I consider that pretty valuable.Probably I’m more excited about, at the moment, about pushing on our current list promising proposals. Since I spent a bunch of time thinking about alternatives and it doesn’t seem as great to me. But I also think there’s a lot of value to clarifying our understanding of the problem. More like trying to generate totally different proposals. Trying to understand what the possibilities are like.


Prompt: Great, yeah. Well we’re planning to get someone on from MIRI on in a couple of months time. Perhaps when it fits better with their plans and they’re hoping to hire. So we’ll some synergies between having the podcast and them actually having some jobs available.

Response: That makes sense.


Prompt: So make it a little bit more concrete. What are Open AI’s hiring opportunities at the moment? And, in particular, I heard that you’re not just hiring ML researchers but also looking for engineers. So I was interested to learn kind of how they help with your work and how valuable those roles are compared to the kind of work maybe that you’re doing?

Response: I think there’s a sort of spectrum between … Yeah there’s a spectrum between research and engineering. Or like most people at Open AI don’t sit in either extreme of that spectrum. So most people are doing some combination of thinking about more conceptual issues in ML, and running experiments, and writing code that implements ideas, and them messing with that code thinking about how it works, like debugging.Yeah there’s a lot of steps in this pipeline that are not that cleanly separated. And so, I think there’s value on the current margin from all the points on this spectrum. And I think actually at the moment, right now, I think I’m still spending or the safety team is still spending a reasonably large … Like even people who are nominally very far on the research end are still spending a pretty large fraction of their time doing things that are relatively far towards engineering. So spending a lot of time setting up and running experiments, getting things working.Again, the spectrum between engineering and research is I think not that clean. Or ML is not in really a state where it’s that clean. So I think right now there’s a lot of room for people who are more at the engineering side. I think what I mean by more at the engineering side is people who don’t have a background doing research in ML, but do have a background doing engineering. And who are interested in learning about ML and willing to put in some time on the order of months, maybe. Like getting more experienced thinking about ML, doing engineering related to ML. I think there’s a lot of room for that.Mostly … So I mentioned these three problems. The first problem was actually getting the engineer experienced to make say amplification or debate work at scale. I think that involves a huge amount of getting things to work. Sort of by the construction of the task. And similarly in the third category of trying to push safety out far enough that it’s engaging with … that ML could actually be interacting in an interesting way with human cognition. I think that also involves again pushing things to a relatively large scale. Doing some research or some work that’s more similar to conventional machine learning work rather than being safety in particular.I think both of those problems are pretty important. And both of them require … like are not that heavily weighted towards very conceptual machine learning work. I think my current take, like I currently consider a second category of work as figuring out from a conceptual perspective is this a good scheme to do? Seems like the most important stuff to me, but also seems like very complimentary with the other two categories in a sense of our current philosophy which I’m pretty happy with, is like we actually want to be building new systems and starting to run experiments on them in parallel with thinking about does this scheme … Like what are the biggest conceptual issues for some combination of the experiments can also keel off … Like even if the conceptual stuff work, if the experiments don’t that’s like another reason that the thing can be a non starter. And second, that who can run a bunch of experiments that actually give you a lot of evidence about … Like help you understand the scheme much better.And obviously, independent of the complimentarity. Actually being able to implement these ideas is important. Like there’s obviously complimentarity between knowing whether x works and actually having the expertise there able to implement x. Right? The case that we’re aiming at is the cases where we have both developed a conceptual understanding of how you can build blind AI, but that have actually developed teams and have groups that are understand that and are trained to actually put it into practice in cases where it matters. So we’d like to aim towards the one where there’s a bunch of teams that are able to … Yeah that are basically able to apply cutting edge ML to make AI systems that are aligned rather than unaligned.That’s, again, harking back to the very beginning of our discussion, we talked about these two functions of safety teams. I think the second function of actually make the AI blind is also an important function. Obviously, it only works if you’ve done the conceptual work. But also, the conceptual work is also realistically the main way that’s going to be valuable is if there are teams that are able to put that into practice. And that problem is, to a significant extent, an engineering problem.


Prompt: Just quickly, do you know what vacancies Open AI has at the moment?

Response: So I guess on the safety team, yeah … I mostly think about safety team. On the safety team, we are both very interested in hiring ML researchers who have a background in ML research. Like who have done that kind of work in the past or have done exceptional work in nearby fields and are interested in moving into ML. We’re also pretty interested in hiring ML engineers. That is people who have done engineering work and are maybe interested in learning, or have put in some amount of time. So ideally these are people who are either exceptional at doing engineering related to ML or are exceptional at engineering and have demonstrated that they’re able to get up to speed in ML and are now able to do that quality work.And again, those roles are not … In terms of what they involve, there’s not a clean separation between them. It’s basically just a spectrum. Yeah, there’s several different skills that are useful. We’re really looking for all of those skills. Like the ability to build things, the ability to do engineering, the ability to do those parts of engineering that are distinct to ML. The ability to reason about safety, the ability to reason about ML. Both of those at a conceptual level.So the safety team is currently looking for the entire spectrum of stuff we do. I think that’s probably the case in several other teams within the organization. That is, the organization is large enough, there’s like a bunch of places now that given a particular skill set … Well, again, given any particular skill set on that spectrum, there’s probably a place. The organization overall is not that large. We’re at the scale of 60 full-time people, I think. So there’s still a lot of roles that don’t really exist that much. Like that would at a very large company. But there’s a lot of engineering to be done. There’re a lot of conceptual work to be done. And a lot of the whole space in between those.


Prompt: Yeah. How does work at Open AI compare to DeepMind and other top places that people should have at the forefront of their brains?

Response: You mean in terms of my assessment of impact? Or in terms of the experience day to day?


Prompt: I think in terms of impact, mostly.

Response: Yeah. I don’t think I have a really strong view on this question. I think it depends in significant part on things like where you want to be and which particular people you’re most excited about working with. I guess those are going to be two biggest inputs. Yeah, I think that both teams are doing reasonable work that accelerates safety. Both teams are giving experience implementing things and understanding how you can be integrated into an AI project.I’m optimistic that over the long run, there will be some amount of consolidation of safety work at wherever happens to be the place that is designing the AI systems for which it’s most needed.


Prompt: Awesome. A question that quite a few listeners wrote in with for you was how much people who were concerned about AI alignment should be thinking about moving into computer security in general? And what’s the relationship between computer security and AI safety?

Response: I think it’s worth distinguishing two relationships between security and alignment. Or like two kinds of security research. So one would be security of computer systems that interface with or are affected by AI. So this is kind of like the conventional computer security problem, but now in a world where AI exists. Or maybe you’re like even aren’t focusing on the fact that AI exist and are just thinking about conventional computer security. So that’s one class of problems.There’s a second class of problems which is the security of ML systems themselves. Like to what extent can an ML system be manipulated by an attacker or to what extent does an ML system continue to function appropriately in an environment containing an attacker. So they have different views about those two areas.So on the first area, computer security broadly, I think my current feeling is that computer security is quite similar to other kinds of conflict. So that is, if you live in a world where it’s possible to attack. You know, someone’s running a web server. It’s possible to compromise that web server. Like a bunch of people have computers. It’s possible to effectively steal resources from them or to steal time on their computers. That’s very similar to living in a world where it’s possible to take a gun and shoot people. And I like regret, in general, I love it if there are fewer opportunities for destructive conflict in the world. Like it’s not great if it’s possible to steal stuff or blow stuff up or so on.But from that perspective, I don’t think computer security is like … I think the core problem in the AI alignment, like the core question is can we build AI systems that are effectively representing human interests? And if the answer is no, then there are enough forms of possible conflict that I think we’re pretty screwed in any case. And if the answer is yes, if we can build powerful AI systems that are representing human interests, then I don’t think cyber security is a fundamental problem anymore than the possibility of war as a fundamental problem. Like it’s bad. It’s perhaps extremely bad, but we will able … At that point, the interaction will be between AI systems representing your interests and AI systems representing someone else’s interests or AI systems representing no one’s interest.And, at that point, I think the situation is probably somewhat better than the situation is today. That is I expect cyber security is less of a problem in that world than it is in this world if you manage to solve alignment. So that’s my view on computer security that’s not conventional computer security and how alignment interfaces it with it. I think it can be … I basically think quantitatively, computer security can become somewhat more important during this intermediate period. Or AI is especially good at certain kinds of attacks and maybe not as useful. Like it may end up not being as useful for defense. And so one might want to intervene on making AI systems more useful for defense. But I think that doesn’t have outsize utilitarian impact compared to other cause areas in the world.I think security of ML systems is a somewhat different story. Mostly because I think security of ML systems … like intervening on security of ML systems seems like a very effective way to advance alignment, to me. So if you ask how are alignment problems likely to first materialize in the world? Like supposing that I have built some AI system that isn’t doing exactly the thing that I want. I think the way that that’s likely to first show up is in the context of security.So if I build like a virtual system that’s representing my interests on the internet, it’s like a little bit bad if they’re not exactly aligned with my interests. But in a world containing an attacker, that becomes catastrophically bad often. Because an attacker can take that wedge between the values of that system and my values and they can sort of create situations that exploit that difference.Right. So, for example, if I have an AI that doesn’t care about some particular fact. Like it doesn’t care about the fact that it uses up a little bit of network bandwidth whenever it sends this request. But I would really care about that because I wouldn’t want to keep sending requests arbitrarily. So an attacker can create a situation where my AI is going to become confused and because it isn’t attending to this cost. An attacker is motivated to create a situation where the AI will therefore pay a bunch of the cost. So motivated to trick my AI that doesn’t care about sending messages into sending very very large numbers of messages.Or like if my AI normally behaves well, then there exists this tiny class of inputs with very very small probability and encounters an input that causes it to behave maliciously. And that will appear eventually in the real world, perhaps. And that’s part of, sort of, the … part of the alignment concern. Is that that will appear naturally in the world with small enough probability. Or as you run AI long enough. But it will definitely first appear when an attacker is trying to construct a situation in which my AI behaves poorly. So I think security is this interesting connection where many alignment problems, not literally all, but I think a majority you should expect to appear first as security problems. And as a result, I think security is sort of one of the most natural communities to do this kind of research in.


Prompt: When you say an attacker would try to do these things, what would be their motivation there?

Response: Honestly it would depend on exactly what AI system it is. But a really simple case would be if you have a virtual assistant going out to make purchasing decisions for you. The way it makes its decisions is slightly wrong. There are a thousand agents in the world, there are a thousand people who would like the virtual assistants to send them some money. So if it’s possible to manipulate the decision it uses for deciding where to send money, then that’s a really obvious thing to try and attack.If it’s possible to cause it to leak your information. So suppose you have an AI which has some understanding of what information … of what your preferences are, but doesn’t quite understand exactly how you regard a privacy. There’s ways of leaking information that it doesn’t consider a leak. Because it has an almost, but not completely, correct model of what constitutes leaking. Then an attacker can use that to just take your information by setting up a situation where the AI doesn’t regard something as a leak, but it is a leak.There’s any difference between what is actually bad and what you’re AI considers bad. And an attacker can come in and exploit that difference if there’s some action that would have a cost to you and it benefits the attacker. Then the attacker wants to set things up so that your AI system is not recognizing the cost to you.So taking money, taking information, using your computer to launch other malicious activities. Like run denial of service. Just causing destruction. Like there’s some fraction of attackers who just want to run denial of service stacks. So if you can compromise integrity of a bunch of AI systems people are using, that’s a bummer.Maybe they want to control what content you see. So if you have AI systems that are mediating how you interact with the internet. You know your AI says, here’s something you should read. There are tons of people who would like to change what your AI suggests that you read just because every eyeball is worth a few cents. If you can deploy that at scale, it’s like a lot of cents.So that’s the kind of situation where some of those problems aren’t alignment. There’re a lot of security problems that aren’t alignment problems. But I think it’s the case that many many alignment problems are also security problems. So if one were to be working in security of ML with an eye towards working on those problems that are also alignment problems, I think that’s actually a pretty compelling thing to do from a long term AI safety perspective.


Prompt: So, it seems to me like AI safety is a pretty fragile area where it will be possible to cause harm by doing subpar research or having the wrong opinions or giving the wrong impression, being kind of a loudmouth who has not terribly truth-tracking views. How high do you think the bar is for people going into this field without causing harm? Is it possible to be kind of at the 99th or 99th.9th percentile of suitability for doing this but still on balance: not really do good because the kind of unintentional harm that you do outweighs the positive contribution that you made?

Response: So, I do think the current relationship between the AI alignment community or safety community and the ML community is a little bit strange in that you have this weird situation with a lot of external interests in safety and alignment. We have a lot of external funding, a lot of people on the street. It sort of sounds like a compelling concern to them that causes a lot of people in machine learning to be kind of on the defensive. That is, they see a lot of external interest that’s often kind of off-base or doesn’t totally make sense. They’re concerned about policies that don’t make sense or diversion of interest from issues they consider important to some incoherent concerns.So, that means, again, they’re a little bit on the defensive in some sense, and as a result, I think it’s kind of important for people in the field to be reasonably respectful and not causing trouble, because there’s more likely in most context to actually cause a sort of hostile response. I don’t know if that’s much of a property of people. I think someone who believed that this was an important thing, if you’re at the point where you’re like, “Yep, I’m really concerned about causing political tension or really rocking the boat-”


Prompt: That’s not a good sign.

Response: Yeah, when they hit that point, if you’re at that point and you’re basically behaving sensibly, then I think things could probably be okay. I’ve definitely sometimes … I have from time to time caused some distress or run into people who are pretty antagonistic towards something I was saying, but I mostly think if you care about it a lot and are being sensible, then I’d be like very surprised if the net effect was negative. I think a lot of people don’t care about it very much. They would disagree with this position and say that, “Look, the reason people are antagonistic is not because they’re being reasonably concerned about outsiders who don’t have a clear understanding pushing bad policies.” The reason that they’re defensive is just ’cause they’re being super silly, and so it’s just time for a showdown between people who are being silly and people who have sensible views.And like, if you’re coming in with that kind of perspective, then, presumably this question’s not interesting to you because you’re just like, “Yeah, Paul’s just one of the silly sympathizers.” It’s not clear that I’m allowed to give recommendations to people like that or that they would … it’s not clear that they would be interested in the recommendations. I would recommend just as part of a compromise perspective, if you have that view, then there exists other people like Paul who have a different view on which, like, there are some reasonable concerns everyone wants to behave somewhat respectfully towards those concerns. It’ll be good if we all compromised and just didn’t destroy things or really piss people off.If you couldn’t have worked on AI…


Prompt: So, if we imagine you and your colleagues and people who are kind of similar to you in other organizations before you got in AI safety but you had your skills and talents and interests, but I would say that you can’t work on AI safety, what do you think you should have done otherwise?

Response: Yeah, so by can’t work on AI safety, you mean let us ignore all of the impacts of my work via the effect on … a natural thing that I might have done would be go into AI, and I’m like, “AI seems important independent of alignment. It seems like AI’s reasonably likely,” as a person with a sort of technical background, it kind of seemed, especially in the past … this is more obvious ’cause I’ve neglected this argument in the past. But, it seems like there’s a good ratio of effect of the area to congestion or number of people trying to work on it, and it was a good match for my comparative advantage.


Prompt: Yeah, let’s maybe set that aside as well ’cause it’s pretty similar.

Response: Yeah, that seems not in the spirit of the question. Yeah, so setting aside all of AI … and let’s set aside everything that’s having an effect via the overall importance of AI. I am pretty excited about overall improving human capacity to make good decisions, make good predictions, coordinate well, etc., so I’m pretty excited about that kind of thing. I think it would be a reasonable bet, so that includes both stuff like … some of these things aren’t a good fit for my comparative advantage so it’s probably not what I should do. Examples of things that aren’t a good fit for my comparative advantage are: understanding pharmacological interventions to make people smarter, understanding … just having a better map of determinants of cognitive performance. “How can you quickly measure cognitive performance? What actually determines how well people do at complicated messy tasks in the real world,” so that you can intervene on that.I think that’s an area where science can add a really large amount of value. It’s very, very hard for a firm to add value in that space compared to a scientist ’cause you’re just gonna discover facts and you’re not gonna be able to monetize them very well, probably. That’s an example of proving human capacity in a way that I probably wouldn’t have done because it’s not a great fit for my abilities. Things that are a better fit for my abilities are like stuff that’s more about what sort of institutions or mechanisms do you use? I don’t know if I would have worked on that kind of thing. I might have. So, an example of a thing I might work on is like-


Prompt: A little bit more law and economics or-

Response: Yeah, so an example of a thing that I find very interesting is the use of decision markets for collective decision making. And so, that’s an example of an area that I would seriously consider and I think there’s a lot of very interesting stuff you can do in that space. It’s not an area I’ve thought about a huge amount because it seems like significantly less high-leverage than AI, but it is like a thing which I think there’s a lot more mathematical work to do, and so if you’re avoiding AI and you’re like, “We’re just math, really,” I’m almost certainly going to be working in some area that’s very, very similar to theoretical computer science in terms of what skills it requires.


Prompt: I guess, yeah, are there other key questions in that field that stand out as being particularly important in maths, computer science, other than AI-related things?

Response: So, definitely, most of the questions people ask, I think, are, if they’re relevant at all, primarily relevant through an effect on AI, so I don’t know how much, exactly. I mean, I took those off the table. Maybe that was too much. I think the basic problem is if you really care about differential progress, effective altruists tend to have this focus on, “It doesn’t matter if we get somewhere faster. It mostly matters what order technologies are developed in or what trajectory we’re on.” I think really a lot of the things people work on are like … a lot of things people work on in math or computer science are like founded on this … based on this principal, “We don’t know how X is going to be helpful, but it’s going to be helpful in some way,” which I think is often a valid argument, but I think is not helpful for differential progress, or like, you need a different flavor of that argument if you wanna say … It’s hard to say, “We don’t know how this is going to be helpful, but we believe it’s going to be helpful to things that are good.”


Prompt: Specifically, yeah.

Response: Yeah, so I think a lot of stuff in math and computer science is less appealing from like a long-run altruist perspective because of that. I think stuff on decision making in particular, like, “What kinds of institutions do you …” I think I was very interested in and did work on my thesis was just like, “There’s this giant family of problems. You have N people. They like each have access to some local information and would like to make some decisions. You can formalize this in our problems in that space. They would like to decide what to produce and what to consume and what to build,” so I’m just asking those questions saying, “What are good algorithms that people can use?” So, I really am using the computer science question, yeah. I don’t know that much about these areas, but it’s very exciting kind of area.


Prompt: You may not have anything new to say about this one, but what would you say are the most important ways that people in the effective altruism community are approaching AI issues incorrectly?

Response: So, I think one feature of the effective altruism community is its past dependence on founder effects, or people in EA who are interested in AI safety are often sort of very informed by this mirrored perspective for the very sensible reason the MIRI folk and Nick Bostrom were probably the earliest people talking seriously about the issue. So there’s like the cluster of things that I would regard as errors that would come with that. So like, some perspective on how you should think about sophisticated AI systems, so for example, very often thinking in terms of a system that has been given a goal.This is actually not a mistake that MIRI makes. This is a mistake many EAs. Make. Many EAs would think about an AI as being handed some goal, like an explicit representation of some goal, and the question is just how do we choose that specific representation of a goal such that pursuing it leads to good outcomes, which I think is an okay model of AI to work with sometimes, but it’s mostly not … certainly not a super accurate model, and most of the problems in AI alignment appear in that model. So, that’s like a kind of error … again, attributing that one to MIRI is somewhat unfair and that MIRI themselves wouldn’t make this error, but is a consequence of people-


Prompt: It’s kind of a bastardized version of their view.

Response: That’s right. An analogous thing is: I think that the way you should be thinking, probably, about building safe AI systems is more based on this idea of corrigibility, ’cause there’s AI systems that are going along with what … helping people correct them, helping humans understand what they’re doing and overall, participating in a process that points in the right direction rather than attempting the communicate the actual, “What is valuable?” or having an AI system that embodies what humans intrinsically want in the long-run.So, I think that’s a somewhat important distinction and that’s kind of intuitively, if an ML person talks about this problem, they’re really going to be thinking about from that angle. They’re gonna be seen like, “Great, we want our AI to not kill everyone. We want it to help us understand what’s going on,” etc., and so sometimes EAs can relate the perspective of, “But consider the whole complexity of moral value and how would you communicate that to an AI?” I think that is like an example of a mismatch that’s probably mostly due to an error on the EA side, though it’s certainly the case that this concept, corrigibility’s a complicated concept and if you actually think about the mechanics of how that works, it’s like, really, there’s a lot more moving parts than the normal ML perspective kind of suggests.Or like, again, it’s not even really like current ML perspective. It’s like the knee jerk response of someone who’s been actually thinking about ML systems. I guess I have difference of views with … I think EAs often have maybe also founder factory reasons like … actually no, I think for complicated reasons, they tend to have a view where the development of AI is likely to be associated with both sort of very rapid changes and also very rapid concentration of power. I think the EAs overestimate the extent to which or the probability of that happening, so this is like, yeah, that’s certainly a disagreement between me and most EAs. I think it’s much more likely we’re gonna be in the regime where there’s reasonably broadly distributed AI progress and ais getting deployed a whole bunch all around the world.And, maybe that happens rapidly over the timescale of a year or two years that the world moves from something kind of comprehensive to something radically alien, but it’s not likely to be a year during which somewhere inside Google, AI’s being developed and at the end of the year, rolls out and takes over the world. It’s more likely to be a year during which just everything is sort of in chaos. The chaos is very broadly distributed chaos as AI gets rolled out.


Prompt: Is it possible that there’ll be better containment of the intellectual property such that other groups can’t copy and one group does go substantially ahead? At the moment, almost all AI research is published publicly such that it’s relatively easy to replicate, but that may not remain the case.

Response: Yeah, so I think there’s definitely this naïve economic perspective on which this would be incredibly surprising, namely if … so, in this scenario where AI’s about to take over the world, then … and it’s driven primarily by progress in the AI technology rather than controlled large amounts of hardware, then that intellectual property now … the market value, sort of market to market would be like ten trillion dollars or whatever, so you sort of expect an actor who is developing that … the amount of pressure, competition to develop that would be very large. You expect very large coalitions to be in the lead over small actors.So, it wouldn’t … Google’s not quite at the scale where they could plausibly do it. You could imagine sort of if all of Google was involved in this project, that becomes plausible, but then again, you’re not imagining a small group in a basement. You’re imagining an entity, which was already producing on the order of like … was already valued on the order of multiple trillions of dollars taking some large share of its resources into this development project.And, that’s kind of conceivable. The value of Google going from five trillion dollars to a hundred trillion dollars, that’s a huge jump. It’s 20x in value, a hundred trillion dollars being your value if you take over the world. The 20x is a huge jump, but that’s kind of in the regime of what’s possible, whereas I think a billion dollars taking over the world is just super implausible. There’s an economic perspective which makes that prediction very confidently. To compare that to the real world, you have to think about a lot of ways in which the real world is not like an idealized simple economic system, but I still think it will be the case that probably AI development will involve very large coalitions involving very large amounts of hardware, large numbers of researchers regardless of if intellectual property is contained really well, then it might take place within a firm or a tightly coordinated cluster of firms rather than distributed across the academic community.In fact, I would not be at all surprised if the academic community didn’t a super large role, but then the distinction is between distributed across a large number of loosely coordinated firms vs distributed across a network of tightly coordinated firms, and like, in both cases, it’s a big group. It’s not like a small group being covert. And, once you’re in the regime of that big group, then yeah, probably what ends up happening there is like the price … so, if it’s like Google’s doing this, unless they’re in addition to being really tight about IP, also really tight about what they’re doing, you see the share price of Google start growing very, very rapidly in that world, and then probably, as that happens, eventually you start running into problems where you can’t scale markets gracefully and then policymakers probably become involved.At the point in the market is staying, Google is roughly as valuable as everything else in the world. Everyone is like, “Geez, this is some serious shit.” Google’s an interesting case, actually, ’cause corporate governance at Google is pretty poor, so Google has this interesting property where it’s not clear that owning a share of Google would actually entitle you to anything if Google were to take over the world. Many companies are somewhat better governed than Google in this respect.


Prompt: Explain that.

Response: So, Google is sort of famous for shareholders having very little influence on what Google does, so if Google hypothetically were to have this massive windfall, it’s not really clear … it would be kind of a complicated question what Google as an organization ends up doing with that windfall, and Google seems kind of cool. I like Google. They seem nice, probably like they’d do something good with it, but it’s not obvious to me that being a shareholder in Google then gives you-


Prompt: You don’t get the dividend? You could sell the shares or-

Response: You get the dividend, but it’s not clear whether it would be a dividend. For example, most shares that are sold on Google-


Prompt: You’re saying there’s a possibility of retaining the earnings to just invest in other things and it never gets handed back-

Response: Yeah, they’d build some Google city, more Google projects.


Prompt: Interesting.

Response: In particular, most shares of Google that are traded are non-voting shares, I think. I don’t actually know very much about Google’s corporate governance. They’re sort of famous-


Prompt: There’s two classes, I think, yeah.

Response: So, I believe a majority of voting shares are still held by like three individuals. So, I think the shareholders don’t have any formal power in the case of Google, essentially. There’s a question of informally, there’s some expectations, and again, if you’re taking over the world, formal mechanisms are probably already breaking down.


Prompt: There’s also plenty of surplus to distribute.

Response: Well, yeah, that depends on what you care about. So, from the perspective, in general, like, as AI’s developed, from the perspective of humans living happy lives, there’s sort of massive amounts of surplus. People have tons of resources. From the perspective of if what you care about is relative position or owning some large share of what is ultimately out there in the universe, then there’s, in some sense, there’s only one universe to go around, and so people will be divvying it up.So, I think the people who are most interested in living happy lives and having awesome stuff happen to them and having their friends and family all be super happy, those people are all just gonna be really satisfied and it’s gonna be awesome, and the remaining conflict will be amongst either people who are very sort of greedy in the sense that they just want as much stuff as they can have, or states that are very interested in insuring the relative prominence of their state, things like that. Utilitarians, I guess, are one of the offenders here. A utilitarian wouldn’t be like, “Great, I got to live a happy life.” Utilitarians like-


Prompt: They have linear returns to more resources, more than most people do, yeah. I guess any universalist moral system may well have this property, or actually not necessarily, but most of them.

Response: Yeah, I think a lot of impartial values generally have, yeah.


Prompt: Another blog post you wrote recently was about how valuable it would be if we could create an AI that didn’t seem value aligned, whether that would have any value at all or whether it would basically mean that we’d gotten 0 value out of the world. Do you want to explain what your argument was then?

Response: Yeah, so I think this is a perspective that’s reasonably common in the ML community and in the broader academic world or broader intellectual world, namely, you build some very sophisticated system. One thing you could try and do is you could try and make it just want what humans want. Another thing you could do is you could just say, “Great, it’s some very smart system that has all kinds of complicated drives. Maybe it should just do its own thing. Maybe we should be happy for it, you know, the same way that we think humans are an improvement over bacteria, we should think that this AI we built is an improvement over humans.”


Prompt: Should live its best life.

Response: Yeah, so I think it’s not an uncommon perspective. I think people in the alignment community are often pretty dismissive of that perspective. I think it’s a really hard. I think people on both sides, both people who sort of accept that perspective intuitively and people who dismiss that perspective, I think haven’t really engaged with how hard a moral question that is. Yeah, I consider it extremely not obvious. I am not happy about the prospect of building such an AI just ’cause it’s kind of an irreversible decision, or handing off the worlds to this kind of AI we built somewhat irreversible decision.


Prompt: It seems unlikely to be optimal, right?

Response: Yes. I guess I would say half as good. If it’s half as good as humans doing their thing, I’m not super excited about that. That’s just half as bad as extinction. Again, trying to avoid that outcome, it’d be half as important as trying to avoid extinction, but again, the factor of two’s not going to be decisive. I think the main interesting question is, “Is there such an AI you could build that would be close to optimal?” And I do agree that a priori, most things aren’t going to be close to optimal. It’d be kind of surprising if that was the case. I do think there are some kinds of AIs that are very inhuman for which it is close to optimal, and understanding that border between when that’s very good, like when we should, as part of being a cosmic citizen, should be happy to just build the AI vs when that’s a great tragedy.It’s important to understand that boundary if there’s some kind of AI you can build that’s not aligned that’s still good, so in that post I both made some arguments for why there should be some kinds of AIs that are good despite not being aligned, and then I also tried to push back a little bit against the intuitive picture some people have that is the default.


Prompt: Yeah, so I guess the intuitive picture in favor is just, “It’s good when agents get what they want and this AI will want some things, and then it will go about getting them, and that’s all for the good,” and the alternative view would be, “Well, yes, but it might exterminate life on earth, and then fill the universe with something like paperclips or some random thing that doesn’t seem to us like it’s valuable at all, so what a complete waste that would be.” Is that about right?

Response: That’s definitely a rough first pass. That’s basically right. There’s definitely a lot that can be said on the topic. For example, someone who has the favorable view could say, “Yes, it would be possible to construct an agent which wanted a bunch of paperclips, but such an agent would be unlikely to be produced, so you’d have to go out of your way.” In fact, maybe the only way to produce such an agent is if you’re really trying to solve alignment. If you’re just trying to run something like evolution, then consider the analogy of evolution: humans are so far from the kind of thing that, yeah.So, one position would be, “Yeah, there exists such bad AIs,” but if you run something like evolution, you’ll get a good AI instead. So, that perspective might then be optimistic about the trajectory of modern ML that is, on some alignment perspectives, you’re like, “Well, this is really terrifying. We’re just doing this black box optimization. Who knows what we’re going to get?” From some other perspectives, you’re like, “Well, that’s what produced humans, so we should pay it forward.” I think, also, people get a lot of mileage out of the normal analogy to descendens. That is, people say, “Well, we would have been unhappy had our ancestors been really excited about controlling the trajectory of our society and tried to ensure their values were imposed on the whole future,” and likewise, even if our relationship to AI systems we built is different than the relationship of our ancestors to us, it has this structural similarity and likewise, the AI would be annoyed if we went really out of our way and paid large costs to constrain the future structure of civilization, so maybe should be nice and do unto others as we would have them do unto us.


Prompt: I don’t find that persuasive, personally.

Response: I certainly don’t find it persuasive out of the box, yeah.


Prompt: It just seems very different. I guess, were very similar by design to humans from 500 years ago just with probably more information and more time to think about what we want, whereas I think you can’t just … yeah, an AI might just be so differently designed that it’s like a completely different jump, whereas from our point of view, it could be, “Well, yeah.”

Response: I think the more compelling … so, I don’t really lean much on this. I don’t take much away from the analogy to descendens. I think it’s a reasonable analogy to have thought about, but it’s not going to run much of the argument. I think the main reason that you might end up being altruistic towards, say, the kind of product of evolution would be if you said, “From behind the veil of ignorance, humans have some complicated set of drives, etc.” If humans go on controlling earth, if that set of values and preferences humans have is gonna get satisfied, if we were to run some new process that’s similar to evolution, it would produce a different agent with a different set of values and preferences, but from behind the veil of ignorance, it’s just as likely that our preferences would be the preferences of the thing that actually evolves on earth is that our set of preferences would be the preferences of this AI that got created. So, if you’re willing to step far enough back behind the veil of ignorance, then you might say, “Okay, I guess.”


Response: And, I think there’s come conditions under which you can make that argument tight and so even a causal of perfectly selfish causal decision theorist would in fact for like these normal, weird acausal trade reasons would in fact want to let the AI … would be happy for the AI, and the question of: outside of those very extreme cases where there’s a really tight argument, that you should be happy, how happy should you be if there’s a loose analogy between the process that you ran and biological evolution?


Prompt: So, what do you think are kind of the best arguments both for and against thinking that an unaligned or what seems like an unaligned AI would be morally valuable?

Response: So, I think it certainly depends on which kind of unaligned AI we’re talking about, and so one question is, “What are the best arguments that there exists an aligned AI which is morally valuable?” And another question is like, “What are the best arguments that a random AI is morally valuable?” Etc. So, I guess the best argument for the existence, which I think is an important place to get started, or if you’re starting from this dismissive perspective like most people in the alignment community have intuitively, I think the existence argument is a really important first step. I think the strongest argument on the existence perspective is: consider the hypothetical where you’re actually able to, in your computer, create a nice little simulated planet from exactly the same distribution as Earth, so you run Earth, you run evolution on it, you get something very different from human evolution, but it’s exactly drawn from the same distribution.


Prompt: You’d think it’s like 50/50 whether it’s likely to be better or worse than us, on average, right?

Response: Well, from our values, it might be … having conditioning now on our values, it might definitely be much worse.


Prompt: But conditioning of being agnostic about what values are good?

Response: That’s right, or yeah, it’s a really complicated moral philosophy question. Then, the extreme … I think we could even make it actually tighter. So, if you were to just make such a species and then let that go in the universe, I think then you have a very hard question about whether that’s a good deal or a bad deal. I think you can do something a little bit better. You can do something a little bit more clearly optimal, which is like if you’re able to create many such simulations, run evolution not just once, but many times, look across all of the resulting civilizations and pick out a civilization which is constituted such that it’s going to do exactly the same thing you’re currently doing, such that when they have a conversation like this, they’re like, “Yeah, sure. Let’s let out that … let’s just run evolution and let that thing prosper,” then kind of like now, the civilizations who follow the strategy are just engaged in this musical chairs game where each of them started off evolving on some worlds, and then they randomly simulate a different one of them in the same distribution, and then that takes over that world.So, you have exactly the same set of values in the universe now across the people who adopt this policy, just shuffled around. So, it’s clear that it’s better for them to do that than it is for them to say, face some substantial risk of building an aligned AI.


Prompt: Okay, so I didn’t understand this in the post, but now I think I do. The idea is that: imagine that there’s a million universes, all with different versions of Earth where life as evolved.

Response: If you’re willing to go for a really big universe, you can imagine they’re literally just all copies of exactly the same solar system on which evolution went a little bit differently.


Prompt: And so they all end up with somewhat different values, and you’re saying … but if they’re all … if all of their values imply that they should just reshuffle their values and run a simulation and then be just as happy to go with whatever that spits out is what they seem to prefer, then all they do is kind of trade places on average. They all just … you all just end up with different draws from this broad distribution of possible values that people can have across this somewhat narrow but still broad set of worlds? But, you’re saying this is better because they don’t have to worry so much about alignment? Oh, you mean why are things better after having played … after having. Yeah, why does the musical chairs thing where everyone just flips values on average with other people produce a better outcome in total?

Response: Yeah, so I think this is most directly relevant as an answer to the question, “Why should we believe there exists a kind of AI that we would be as happy to build as an aligned AI even though it’s unaligned?” But in terms of why it would actually be good to have done this: the natural reason is, we have some computers. The concerning feature of our current situation is that human brains are not super … we have all these humans. We’re concerned that AIs running on these computers are going to be better than humans such that we’re sort of necessarily going to have to pass control over the world off to things running on computers.So, after you’ve played this game musical chairs, now the new residents of our world are actually running on the computers, so now, as if you got your good brain emulations for free, that is now, those people who have access to simulations of their brain can do whatever it is they would … whatever you would have done with your AI, they can do with themselves. Yeah, there’s really a lot of moving parts here and a lot of ways this maybe doesn’t make any sense.


Prompt: Okay, let me just … so, if we handed it off, if we handed off the future to an AI that was running a simulation of these worlds and using that as its reference point for what it should value, on average, from this very abstracted point of view, this would be no worse, and, if all of the people in this broad set did this, then they would save a bunch of trouble trying to get the AI to do exactly what they want in that universe. They could all just kind of trade with one another, or they all get to save the overhead of trying to make the AI align with them specifically, and so they have to align it to some other pole that they’ve created of, yeah, some evolutionary process that listens to inside the computer?

Response: And the concern is presumably not the overhead, but rather the risk of failure. That is, if you think there’s a substantial risk you would build the kind of AI which is not valuable, then this would be … that’s our current state. We might build an AI that does something no one wants. We could instead build an AI that does something that we want. Maybe a second, a third alternative, which is the same as the good outcome between those two is just build an AI that reflects values that are the same, from the same distribution of values that we have.


Prompt: Okay, so you try to align it with your values, and if you fail, I’d think, “Well, there’s always this backup option and maybe it’ll be valuable anyway.”

Response: This is definitely plan B, and so it’d mostly be relevant … and again, to be clear, this weird thing with evolution is not something that’s going to get run because you can’t sample from exactly the same distribution as evolution. It would just prompt the question, “What class of AIs have this desirable future?” given that you believe at least one does, and yeah, it would be a plan B.So the reason to work on this moral question, “What class of AIs are we happy with despite not being aligned with us?” And the reason to work on that moral question would be that if you had a reasonable answer, that … it’s an alternative to doing alignment. If we had a really clear answer to that question, then we could be okay anyway even if we mess up alignment.


Prompt: Okay, so this would be a … yeah, I see. It would be an alternative approach to getting something that’s valuable even if it’s not aligned in some narrow sense with us.

Response: Yeah.


Prompt: And it might be an easier problem to solve, perhaps.

Response: That’s right. At least, people have not … on my list of moral philosophy problems, it’s like my top rated moral philosophy problem. I think not that many people have worked on it that long.


Prompt: So, if you were a moral realist, you’d just believe that there are objective moral facts.

Response: They should be totally fine with this kind of thing. From their perspective, why think that humans are better at discovering objective moral facts than … actually, I don’t know moral realist positions very well, but my understanding, some moral realists would go for that.


Prompt: But, I guess they might look at humans and say, “Well, I do just think that we’ve done better than average, or better than you would expect at doing this.” For example, “We care about this problem to begin with, whereas many other agents just might not even have the concept of morality,” so in that sense, we’re in the top half: maybe not the very top, but I wouldn’t roll the dice completely again, but then it seems like they should also then think that there’s a decent chance. If we did okay, it suggest that there’s a decent chance that if you roll the dice again, you’d get something somewhat valuable, because it would be an extraordinary coincidence if we managed to do really quite well at moralism, figuring what these moral facts are if it was extremely improbable for that to happen to begin with.

Response: Yeah, I mean, it’s definitely: if you’re a moral realist, you’re going to have different views on this question. It’s going to depend a lot on what other views you take on a bunch of related questions. I’m not super familiar with coherent moral realist perspectives, but on my kind of perspective, if you make some moral errors early in history, it’s not a big deal as long as you are on sort of the right path for the spectrumal deliberation, so you might think from the realist perspective, there’d be a big range of acceptable outcomes and you could in fact be quite a bit worse than humans as long as you were again, on this right path of spectral deliberation. I don’t quite know how moral realists feel about deliberation. Would they say there’s a broad … yeah, I think there’s probably a lot of disagreement amongst moralists and it’s just not a-


Prompt: But then if you’re a total subjectivist, do you think there’s nothing that people ought to think is right? Instead, you just kind of want what you want? Why do you care at all about what other people in different hypothetical runs of evolutions would care about? Wouldn’t you just be like, completely, well, “I don’t even care what you want. All I care is about what I individually want and I just wanna maximize that.”

Response: Yeah, and so then you get into these decision theoretic reason to behave kindly. The simplest pass would be, from behind, if you could have made a commitment before learning your values to act in a certain way, then that would have benefited your values and expectation. So, similarly, if there are logical correlations between a decision and the decisions of others with different values, then that might be fine. Even on your values, it might be correct for you to make this decision because it correlates with this other decision, like a decision made by others.In the most extreme case … at some point, I should caveat this entire last however many, 10 minutes, 15 minutes of discussion as like, “This is a bunch of weird shit. It’s doesn’t reflect my behaviour as an employee of open AI like I do normal stuff making AI good for humans.” Anyway, then you get into weird shit where, once you’re doing this musical chairs game, then one step of that was you ran a bunch of simulations and saw which ones you were inclined to participate in the scheme you’re currently running, and so from that perspective, us as humans, we’d be like, “Well, we might as well be in such a simulation,” in which case, even on our neural values, by running this scheme, we’re going to be the ones chosen to take over the outside world.


Prompt: Why are you more likely to be chosen to … or go into the outside world if you’re cooperative?

Response: So, the scheme which would run, if you wanted to do the musical chairs thing, you can’t just simulate a random species and let it take your place because that is then just gonna move from those species that run this procedure, they’re all gonna give up their seat, and then the random species are gonna replace them.


Prompt: So you end up … it’s like evolutionarily a bad strategy.

Response: That’s a bad strategy. The thing that might be an okay strategy is you run the scheme and then you test for each species before you let them replace you. “Did they also run the scheme, and then if so-”


Prompt: Choose from the cooperative ones, yeah.

Response: And then that would cause the incentives to be-


Prompt: Yeah, I think this does get a bit weird once we’re talking about the simulations.

Response: Oh, it’s super weird. It’s super weird. I think the earlier parts were more normal.


Prompt: Yeah, the question of just whether an AI would be morally valuable seems much more mainstream.

Response: I agree. I think it’s also more important. I think this weird stuff with simulations probably doesn’t matter whereas I think the question morally, “How valuable is it to have this AI which has values that are kind of like from some similar distribution to our values?” I think that’s actually a pretty important … I think it’s relatively common for people to think that would be valuable, and it’s not something alignment people have engaged with that much. It’s not a question, to my knowledge, that moral philosophers have engaged with that much: a little bit, but like not … I guess maybe they come from a different perspective than I would want to really tackle the question from, as is often the case with moral philosophers.I guess another point is that I’m also kind of scared of this entire topic in that I think a reasonably likely way that AI being unaligned ends up looking in practice is like: people build a bunch of AI systems. They’re extremely persuasive and personable because we’ve optimized them to … they can be optimized effectively for having whatever superficial properties you want, so you’d live in a world with just a ton of AI systems that want random garbage, but they look really sympathetic and they’re making really great pleas. They’re like, “Really, this is incredibly inhumane. They’re killing us after this or they’re selecting us to … imposing your values on us.”And then, I expect … I think the current way overall, as actual consensus goes is like really, to be much more concerned about people being bigoted or failing to respect the rights of AI systems than to be concerned the actual character of those systems. I think it’s a pretty likely failure more than something I’m concerned about.


Prompt: Interesting. I hadn’t really thought about that scenario. So, the idea is here: we create a bunch of AIs and then we kind of have an AI justice movement that gives AIs maybe more control, like more control over their world and more moral consideration. Then it turns out that while they’re very persuasive at advocating for their moral interests, in fact, their moral interests are, when they’re given moral autonomy, are nothing like ours, or much less than they seem.

Response: Then we’re back to this question which was unclear of how valuable … maybe that’s fine. I don’t actually have a super strong view on that question. I think in expectation, I’m not super happy about it.


Prompt: But by kind of arguing for the moral rights of AIs, you’re making the scenario more possible?

Response: I mostly think it’s gonna be … I strongly suspect there’s going to be serious discussion about this in any case, and I would prefer that there would be some actual figuring out what the correct answer is prior to becoming an emotionally charged or politically charged issue. I’m not super confident, to be clear, about anything we’re saying here. These are not like 80% views. These are 40% views. An example would be like: often when we talk about failure scenarios, I will talk about: there are a bunch of automated autonomous corporations that control resources, and they’re mastering resources that no human gets to choose for any purpose, and people’s response is like, “Well, that’s absurd. Legally, you’re just a machine. You have no right to own things. We’re gonna take your stuff.”That’s something that I don’t is that likely to happen. I suspect that to the extent lots of resources are controlled by AI systems, those AI systems will be, in the interest of preserving those resources, will make fairly compelling appeals for respecting their rights in the same way a human would if you were like … if all humans get around and, “Yeah, we’re just gonna take.” Just such terrible optics. It seems like so much not a thing that I expect our society to do, everyone just being like, “We’re going to take all of these actor’s resources. We just don’t think they have the right to self-determination.”


Prompt: Interesting. It seems like the default to me, but maybe not. I guess the issue is that the AIs would be able to advocate for themselves without human assistance potentially in a way that a corporation can’t. A corporation is still made of people. Do corporations make an argument that, “I’m a separate entity and I deserve rights and should be able to amass resources that don’t go to shareholders?” The problem is there it’s controlled by shareholders so it ultimately bottoms out at people in some way, and AI doesn’t necessarily?

Response: I think it’s both the case that corporations do in fact have a level of rights that would be sufficient to run the risk argument so that if the outcome is the same as corporations, that would be sufficient to be concerned, but I also think that corporations are both … yeah, they do bottom out with people in a way that these entities wouldn’t, and that’s one of the main problems, and also they’re just not able to make persuasive arguments. That is, one, they’re not able to represent themselves well. They don’t have a nice ability to articulate eloquent arguments that could plausibly originate with this actual moral patient making the arguments.And then two, the actual moral case is more straightforward for corporations, whereas I think for AI’s there will actually be a huge amount of ambiguity. I think the sort of default, again from if you interact with people who think about these issues, some right now, if you talk to random academics who think about philosophy and AI or look at Hollywood movies that are somewhat less horrifying than Terminator, I think the normal thing would be, “Yeah, by default, we expect once such agents are as sophisticated as humans, they are deserving of moral considerations for the same kinds of reason humans are,” and it’s reasonably likely that people will deny them that moral consideration, but that would be like a terrible moral mistake.I think that’s like kind of the normal … not normal view, but that’s like if I were to try and guess where the community is heading or well it would end up, that would be my guess.


Prompt: Yeah, I guess I feel like AI’s probably would deserve moral consideration.

Response: I also agree with that, yes. That’s what makes the situation so tricky.


Prompt: That’s true, but then there’s this question of: they deserve moral consideration as to their … I suppose ’cause I’m sympathetic to hedonism, I care about their welfare-

Response: As do I. To be clear, I totally care about their welfare.


Prompt: As do I, yeah, as we all should, but I don’t necessarily then want them to be able to do everything … do whatever they want with other resources which is an … no, but I feel that way about other people as well, necessarily, right, that I want other people on Earth to have high levels of welfare, but that doesn’t necessarily mean I wanna hand over the universe to whatever they want.I just think it makes the character of this debate a lot more contentious. If you’re like, “Yeah, everyone agrees that there’s this giant class of individuals which is potentially reasonably large, which currently does some large fraction of labor in the world, which is asking for the right to self-determination and control of property and so on, and are also way more eloquent than we are,” it’s like, geez.We’ll give you the welfare that we think you should deserve, yeah.

Response: It doesn’t sound good.


Prompt: Yeah.

Response: I mean the main reason I think it’s plausible is we do observe this kind of thing with nonhuman animals. People are pretty happy til you’re pretty terrible to nonhuman animals. I think that-


Prompt: But that’s another case where it’s like, for example, I think that we should be concerned about the welfare of pigs and make pigs’ lives good, but I wouldn’t then give pigs lots of GDP to organize in the way that pigs want, but the disanalogy there is that we think we’re more intelligent and have better values than pigs whereas it’s less clear that’d be true with AI. But, in as much as I worry that AI wouldn’t have good values, it actually is quite analogous, that.

Response: Yeah, I think your position is somewhat … the arguments you’re willing to make here are somewhat unusual amongst humans, probably. I think most humans have more of a tight coupling between moral concern and thinking that a thing deserves liberty and self-determination and stuff like that.


Prompt: And generalize, right. Do you think that they’re bad arguments? It flows more naturally from a hedonistic point of view than a preference utilitarian point of view. That seems to be maybe where we’re coming apart.

Response: Oh, no, I mean I also would be like, “Yep, I care about the welfare of lots of agents who I believe …” I believe it’s a terrible bad thing, but maybe not the worst thing ever if you’re mistreating a bunch of AI systems, ’cause I think they probably are at some point to be moral patients, but I would totally agree with you, though, I could have that position and simultaneously believe that it was like either a terrible moral error to bring such beings into existence or a terrible moral error to give them greater authority over what happens in the world.So, I think that’s a likely place for us to end up in, and I think the level of veer and carefulness in public discussion is not such that those kinds of things get pulled apart. I think it probably mostly gets collapsed into a general rah or boo or … I don’t know that much about how public opinion works, but I’d be happy to take simple bets on this.


Prompt: Well, there’s some selfish reasons why people would not necessarily give large amounts of GDP. You could imagine there’s groups that would say, “Well, we still want to own AIs, but we should treat them humanely.” I guess that doesn’t sound too good now that I say that out loud.

Response: I don’t think it’s gonna play well.


Prompt: It’s not gonna play, yeah.

Response: Also, I mean, there’s just such a strong concentrated interest that is like … most of the cases where this goes badly are cases where there’s a large power imbalance, but in the case we’re talking about, the most effective lobbyists will be AI systems, and it’s going to be this very concentrated powerful interest which cares a lot about this issue, has a plausible moral claim, looks really appealing. It seems kind of overdetermined, basically.


Prompt: Yeah, okay.

Response: This isn’t super important. This is mostly relevant, again, when people say things like, “No, it’s kind of crazy to imagine AI’s owning resources and doing their own thing.”


Prompt: Owning resources themselves, yeah.

Response: And I think that is the default outcome, barring some sort of surprising developments and-


Prompt: Okay, I’ve barely thought about this issue at all, to be honest, which perhaps is an oversight, but I need to think about it some more and then maybe we can talk about it again.

Response: I don’t think it’s that important an issue, mostly. I think, but like, details of how to make alignment work, etc., are more important. I just try to justify them by the additional argument that like, to the extent that you care about what these AI systems want, you really would like to create AI systems that are on the same page as humans. If you get to create a whole bunch of extra new agents, it could be great if you create a bunch of agents whose preferences are well-aligned with the existing agents, and it could be like you just create a ton of unnecessary conflict and suffering if you create a ton of agents who want very different things.


Prompt: Okay. So we’re almost out of time. But just a final few questions. So you’re not only working in this area, but you’re also a donor and you’re trying to support projects that you think will contribute to AI alignment. But it’s an area where there’s a lot of people trying to do that. There’s perhaps more money than people who can usefully take it. So I hear it’s somewhat challenging to find really useful things to fund that aren’t already getting funded. How do you figure out what to fund and would you mind mentioning some of the things that you donate to now?

Response: Yeah. So I think I would like to move towards a world where it’s easier to work on anyone who is equipped to do reasonable AI alignment work. Is able to do that with a minimal hassle. Including if they had differences of view with other organizations currently working in the space. Or if they’re not yet trained up and want to just take some time to think about the area and see if it works out well.I think there are definitely … they’re definitely people who are doing work who are interested in funding and I’ll say not doing crazy stuff. And so one could just, in order to inject more money, dip lower in that. Say look, we like previously … If we’re not really restricted by funding, then our bar ought not be like we’re really convinced this thing is great. Our bar should just be it looks like you’re sort of a sensible person who might eventually figure out what’s … You know, it’s like an important part of personal growth. Maybe this project will end up being good for reasons we don’t understand.So one can certainly dip more in that direction. I think that’s not all used up. I guess the stuff I funded in AI safety over the last year has been … The biggest thing was funding Ought. The next biggest was running this sort of open call for individuals working on alignment outside of any organization. Which has funded three or four people. Actually, I guess most recently, there’s a a group working on IRL under weaker rationality assumptions in Europe. And also supporting — Zvi Mowshowitz and Vladimir Slepnev are running an AI alignment prize, which I’m funding and a little bit involved in judging.


Prompt: Do you think other donors that are earning to give could find similarly promising projects if they looked around actively?

Response: I think it’s currently pretty hard in AI alignment. I think there’s potentially room for … Right, so I think it’s conceivable existing funders including me are being too conservative in some respects. And like you could just say look, I really don’t know if x is good. But there is a probable story where this thing is good. Or ensuring that the many people in the field had enough money that they could regrant effectively. Like many people in the conventional AI safety crowd say if they had enough money, they could regrant effectively and could do whatever they wanted.Yeah, unless you’re willing to get a little bit crazy, it’s pretty hard. I guess it also depends on what you’re … Yeah I think it depends a lot on what your bar is. I think if AI is in fact … Like if we’re on short timelines, then the AI interventions are still pretty good compared to other opportunities. And there might be some qualitative sense of this kind of feels like a longer shot or wackier thing than I would fund in most areas.So I think a donor probably has to be somewhat comfortable with that. Yeah there’s also some claims … Like I think MIRI can always use more money. I think there are some other organizations that can also use more money and it’s not something that I think about that much. In general, giving is not something I’ve been thinking about that much because I think it’s just a lot. It seems much better for me to personally be working on getting stuff done.