Prompt: A few weeks ago, we published our conversation with Pushmeet Kohli who’s an AI robustness and reliability researcher at DeepMind over in London. I guess to heavily summarize Pushmeet’s views, I think he might’ve made a couple of key claims.One was that alignment and robustness issues and his view appear everywhere throughout the development of machine learning systems, so they require some degree of attention from everyone who’s working in the field and according to Pushmeet, this makes the distinction between safety research and non-safety research somewhat vague and blurry and he thinks people who are working on capabilities are also helping with safety and improving reliability also improves capabilities for you because you can then you can actually design algorithms that do what you want.Secondly, I think he thought that an important part of reliability and robustness is going to be trying to faithfully communicate our desires to machine learning algorithms and this is analogous, although a harder instance of the challenge of just communicating with other people, getting them to really understand what we mean. Although of course it’s easy to do that with other humans than with other animals or machine learning algorithms.A third point, was, I guess just a general sense of optimism that DeepMind is working on this issue quite a lot and are keen to hire more people to work on these problems and I guess they sense that probably we’re going to be able to gradually fix these problems with AI alignment as we go along and machine learning algorithms, will get more influential. I know you haven’t had a chance to listen to the whole interview, but you skimmed over the transcript. Firstly, where do you think Pushmeet is getting things right? Where do you agree?

Response: I certainly agree that there’s this tight linkage between getting AI systems to do what we want and making them more capable. I agree with the basic optimism that people will need to address in getting assistance to tackle this ‘do what we want ‘problem. I think it is more likely than not that people will have a good solution to that problem. I think even if you didn’t have sort of long termist, maybe there’s this interesting intervention of, “Should long termists be thinking about that problem in order to increase the probability?”I think even if the actions of the long termist are absent, there’s a reasonably good chance that everything would just be totally fine. In that sense, I’m on board with those claims, definitely. I think that I would disagree a little bit in thinking that there is a meaningful distinction between activities whose main effect is to change the date by which various things become possible activities, whose main effect is to like change the trajectory of development.I think that’s the main distinguishing feature of working on alignment, per se. You care about this differential progress towards being able to build systems the way we want. I think in that perspective, it is the case like the average contribution of AI work is almost by definition zero on that front, because it’s bringing the entire– If you just increased all the AI work by a unit, you’re just bringing everything forward by one unit.I think that doesn’t mean there’s like this well-defined theme which is, “Can we change the trajectory in any way?” and that’s an important problem to think about. I think there’s also a really important distinction between the failure which is most likely to disrupt like the long term trajectory of civilization and the failure which is most likely to be an immediate deal breaker for systems actually being useful or producing money and maybe one way to get at that distinction is related to the second point you mentioned.Communicating your goals to an ML system is very similar to communicating with a human. I think there is a hard problem of communicating your goals to an ML system which we can view as a capabilities problem. Are they able to understand things people say? Are they able to form the internal model that would let them understand what I want or understand– In some sense, it’s very similar to the problem of predicting what Paul would do or it’s a little slice of that problem, like briefing under what conditions Paul would be happy with what you’ve done.That’s most of what we’re dealing with when we’re communicating with someone. We’d be totally happy. If I’m talking with you, I would be like completely happy if I just managed to give you a perfect model of me then the problem is solved. I think that’s a really important AI difficulty for making AI systems actually useful. I think that’s less core to the– That’s less the kind of thing that could end up pushing us in a bad, long-term direction mostly because we’re concerned about the case– We’re concerned about behavior as AI systems become very capable and have a very good understanding of the world around them, of the people they’re interacting with and the really concerning cases are ones where AI systems actually understand quite well, what people would do under various conditions, understand quite well like what they want we think about as normal communication problems between people that are not motivated to act in sort of understand what Paul wants but aren’t trying to help Paul get what he wants and I think that a lot of the interesting difficulty, especially from a very long-term perspective is really making sure that no gaps opens up there.Again, the gap between the problems that are most important in the very long run perspective and the problems that people will most be confronting in order to make AI systems economically valuable, I do think that there’s a lot of overlap, that there are both of these problems that people are working on that make AI systems more valuable and also helping very directly with the long run outcome.I think if you’re interested in differentially changing the trajectory or improving the probability of things going well over the long term, you’re more inclined to focus precisely on those problems, which won’t be essential for making AI systems economically useful in the short term and I think that’s really distinctive to what your motivation is or why you– How you’re picking problems or prioritizing problems.


Prompt: Sounds like you’re a little bit more skeptical of that or do you think ideally people should in the medium term be aiming to work on things that seem they disproportionately push on robustness and reliability?

Response: Yes, I think people who are mostly concerned about the long term trajectory, they face this dilemma in every domain where if you live in the world where you think that almost all of humanity’s problems, almost all of the most serious challenges to humanity are caused by things humans are doing or by things not only the humans are doing, but by things humans are doing that we would often think of as part of productive progress, part of the goal.We’re building new technologies but those technologies are also the things that pose the main risks. Then you have to be picky if you’re a person who wants to change the long term trajectory just because the average– It’s just sort of like I probably am helping address those problems if I go to work and I just go do a random thing. I go work on a random project. Make a random product better.I am helping address the kinds of problems we’re concerned about but I’m also at the same time contributing to bringing those problems closer to us in time. It’s sort of like roughly awash. If you’re on the average product, making the average product work and there are subtle distinctions we could make of like– I think if you are motivated to make products work well, if you’re like, “Not only do I want to do the thing that’s most economically valuable, I want to have more of an emphasis on making this product robust.” I think you’re just generally going to make a bunch of low level decisions that will be helpful. I definitely think that there’s a big– You’re going to have a pretty big impact by being fussy about which problems you work on.


Prompt: I guess there’s this open question of whether we should be happy if AI progress across the board just goes faster. What if yes, we can just speed up the whole thing by 20%. Both all of the safety and capabilities. As far as I understand there’s kind of no consensus on this. People vary quite a bit on how pleased they’d be to see everything speed up in proportion.

Response: Yes. I think that’s right. I think my take which is a reasonably common take, is it doesn’t matter that much from an alignment perspective. Mostly, it will just accelerate the time at which everything happens and there’s some second-order terms that are really hard to reason about like, “How good is it to have more computing hardware available?” Or “How good is it for there to be more or less kinds of other political change happening in the world prior to the development of powerful AI systems?”There’s these higher order questions where people are very uncertain of whether that’s good or bad but I guess my take would be the net effect there is kind of small and the main thing is I think accelerating AI matters much more on the like next 100 years perspective. If you care about welfare of people and animals over the next 100 years, then acceleration of AI looks reasonably good.I think that’s like the main upside. The main upside of faster AI progress is that people are going to be happy over the short term. I think if we care about the long term, it is roughly awash and people could debate whether it’s slightly positive or slightly negative and mostly it’s just accelerating where we’re going.


Prompt: Yes, this has been one of the trickier questions that we’ve tried to answer and in terms of giving people concrete career advice. It seems to me if you’re someone who has done a PhD in ML or is very good at ML, but you currently can’t get a position that seems especially safety-focused or is going to disproportionately affect safety more than capabilities. It is probably still good to take a job that just advances AI in general mostly because you’ll be right in the cutting edge potentially of what’s going on, and improving your career capital a lot and having like relevant understanding of the key issues.The work I guess you think is close to awash. It speeds things up a little bit, everything goes in proportion. It’s not clear whether that’s good or bad but then you can potentially later on go and work on something that’s more alignment specific and that is the dominant term the equation. Does that seem reasonable?

Response: Yes. I think that seems basically right to me. I think there’s some intuitive hesitation with the family of advice that’s like, “You should do this thing, which we think is roughly awash on your values now but there will be some opportunity in the future where you can sort of make a call.” I think there’s some intuitive hesitation about that, but I think that is roughly right. Imagine if you offered Paul, if there were two possible worlds.In one there’s twice as many people working on machine learning and AI but half of them really care about the long term and ensuring that AI has developed in a way that’s good for humanity’s long term and that sounds like a good trade. We maybe then have less chance, less opportunity to do work right now. I think that’s the main negative thing. There will be less time to think about the alignment problem per se but on the other hand, it seems really good if a large fraction of the field really cares about making things go well.I just expect a field that has that character to be much more likely to handle issues in a way that’s good for the long term. I think you can scale that down. It’s easiest for me to imagine the case that for a significant fraction of the field, it is like that, but I think that if anything, like the marginal people at the beginning are having a probably larger, a better cost-benefit analysis for them.


Prompt: I was suggesting that this would be the thing to do if you couldn’t get a job that was like AI alignment specific already. Say that they want to join your team but they’re like just not quite good enough yet, they need to learn more potentially. There’s just only so fast that the team can grow. Even though they’re good, you just can’t hire as quickly as people are coming on board but I suppose you have to make sure if people go into these roles that we think are currently kind of just neutral but good for improving their skills that they don’t forget about that. That the original plan was at some point to switch to something different.There’s a bit of a trap . It seems people just in general, they tend to get stuck in doing what they’re doing now and convince themselves that whatever they’re doing is actually really useful so you might think, “Yes, it would be good to go and then switch out but they might have some doubts about whether in fact he will follow through on that.”

Response: Yes, I think that’s right. It would be an even happier world certainly if you took those half of people who might’ve gone into ML, and you’d instead moved them all into really thinking deeply about the long term and how to make things go well. That sounds like an even better world still. It seems to be pretty good if you really trusted someone to– If someone really cared about the long term, you’re like, “What should I do.” It’s a reasonably good option to just be like, “Go do this thing which is good on the short term and adjacent to an area we think is going to be really important over the long term.”


Prompt: There’s been this argument over the years that it would just be good in some way that we can’t yet anticipate to have people at the cutting edge of machine learning research who are concerned about the long term and alert to safety issues and alert to alignment issues that could play out or could have effects on the very long term. People have gone back and forth on how useful that actually would be to just be in the room where the decisions are getting made.It just occurred to me that it seems the machine learning community is really moving in the direction of sharing the views that you and I hold. A lot of people are just becoming concerned about “Will AI be aligned in the long term?” It might be that if you’re particularly concerned about that now, then maybe that makes you different from your peers right now, but in 10 years’ time or 20 years’ time everyone we’ll have converged on a similar vision as we have a better idea of what machine learning actually looks like and what the risks are when it’s deployed.

Response: Yes, I think that’s an interesting question or an interesting possible concern with that approach. I guess my take would be that there are some I don’t know if you’d call them values differences or deep empirical or worldview differences that are relevant here where I think to the extent that we’re currently thinking about problemsthat are going to become real problems, it’s going to be like much, much more obvious there are real problems.I think that to the extent that some of the problems we think about over the very long term are already obviously problems, people in the ML community are very interested in problems that are obviously problems. Or problems that are affecting the behavior of systems today. Again, if these problems are real, that’s going to become more and more the case over time and some people will become more and more interested in those problems.I still think there are likely to be– There is this question of how much are you interested in making the long term go well versus how much are you doing your job or pursuing something which has a positive impact over the short term, or that you’re passionate about or interested in this other non-long term impact of. I do think there’s just continuously going to be some calls to be made or some different decisions. The field embodies some set of values.I think that people’s empirical views are changing more than the set of implicit values that they have. I think if you just said everyone who really cares about the long term isn’t going into this area, then the overall orientation the field will persistently be different.


Prompt: Do you have any views on the particular technical approaches that Pushmeet mentioned in the episode or that the DeepMind folks have written up on their safety blog?

Response: The stuff I’m most familiar with that Pushmeet’s group is working on is verification for robustness to perturbations. Some work on verification and more broadly and some work on adversarial training and testing. Maybe those are the three things, I don’t know if there’s something else. I’m happy to go through those in order.


Prompt: Yes, go through those.

Response: I guess I’m generally pretty psyched about adversarial testing and training and verification. That is, I think there is this really important problem over both– This is one of those things at the intersection of like it matters over the short term. I think maybe matters even more over the very long term of like you have some AI system, you want to delegate a bunch of work to maybe not just one but a whole bunch of AI systems.If they fail catastrophically, it would be really irrecoverably bad. You can’t really rule out that case with traditional ML training because you’re just going to try a thing out a bunch of cases that you’ve generated so far, experienced so far. You’re really not going to be getting–Your training processes aren’t at all constraining. There’s potential catastrophic failure in any situation that comes up.We just want to have something, we want to change the ML training process to respect– To have some information about what constitutes a catastrophic failure and then not do that. I think that’s a problem that is in common between the short and long term. I think it matters a lot on the long term. It’s a little bit hard to say whether it’s more on the long term or short term, but I care more about the long term..I think that the main approaches we have to that are these– The three I really think about are adversarial training and testing, verification and interpretability or transparency. I just think people getting familiar with those techniques, becoming good at them, thinking about how you would apply them to richer kinds of specifications, how you grapple with these fundamental limitations and adversarial training where you’re like you have to rely on the adversary to think of a kind of case.The way the technique works in general is like, “I’m concerned about my system failing in the future. I’m going to have an adversary who’s going to generate some possible situations under which the system might fail. Then we’re going to run on those and see if it fails catastrophically.” You have this fundamental limitation where adversaries aren’t going to think of everything.It’s like people who are just getting experience, how do we grapple with that limitation? In some sense, verification is like a response to that limitation maybe the space between or when– I think it’s productive to have people thinking about both of verification and the limits of verification and testing and limits of testing. Overall I’m pretty excited about all of that.


Prompt: Do you share Pushmeet’s general optimism?

Response: I don’t know quantitatively exactly how optimistic he is. My guess would be that I’m less optimistic in the sense that I’m like “Well, there’s like 10% chance that we will mess this up and lose the majority of the value of the future.” Whereas that’s not I get when listening to him, it’s not the overall sense I get of where he’s at. It’s a little bit hard to know how to translate between a vibe and an actual level of optimism.


Prompt: Yes, it is interesting. Someone can think there is a 20% chance that we’ll totally destroy everything, but still just a kind of a cheerful disposition. [laughs] I came across this. Well, things go well. Among people working on existential risks and global catastrophic risks, and I guess AI in particular, there’s this trade-off between not wanting to do things that other people disagree with or aren’t enthusiastic about, and at the same time not wanting to have a field that’s so conservative but that there are no experiments done unless there is a consensus behind them. Do you think people are too inclined to make ‘unilateralist curse type’ mistakes or not trying things enough?

Response: I think my answer to this probably varies depending on the area. For reference, I think the policy you want to follow is: is the update on the fact that no one else wants to do this thing and then take that really seriously, engage with it a lot before deciding whether you want to do it. Ideally that’s going to involve engaging with the people who’ve made that decision to understand where they’re coming from.I think I don’t have a very strong general sense of whether we’re more likely to make one mistake or the other. I think I’d expect the world systematically to make too much of the sort of thing can be done unilaterally so it gets done. In the context of this field, I don’t know if there are as many– Yes, I guess I don’t feel bad. I don’t feel super concerned about either failure mode. Maybe I don’t feel that bad about where people are at.


Prompt: The vibe I get in general from the AI policy and strategy people is that they are pretty cautious, quite cautious about what they say and what they do. I guess that’s been a deliberate decision, but I do sometimes wonder whether that’s swung too far in favor of not speaking out enough about their views?

Response: Yes, I guess there are certainly people who have taken, and it’s a diversity of what people do, which I guess is the whole problem. I guess there are definitely people who take a very cautious perspective.


Prompt: I think that they sometimes get a bit cut out of the public discussion, because they’re just not inclined to speak out, which can be a loss at times.

Response: Yes, I definitely think it seems you have a real problem, and you think that a positive part of your channel for impact is communicating your views, but then are very hung up on that or take a strong, shouldn’t communicate views because of unilateralist concerns. I guess in general, on the family of unilateralist concerns I’m least sympathetic to, is probably one intervention is to talk seriously about what kinds of mechanisms might be in place, and how we might respond, if it turned out that AI alignment was hard or if AI progress was rapid. That’s probably the place I’m least, overall sympathetic but I think the cost-benefit looks pretty good on that discussion.


Prompt: Could you say you default towards just on most issues, just even if you’re not taking action, express your true views?

Response: Or at least to the extent there’s useful collaborative cognitive work of thinking about what should be done, how would we respond or what would happen. Being willing to engage in that work as a community, rather than people thinking in private. Maybe taking some care to be well, you don’t want to look say inflammatory stuff, you don’t want to get people really upset but you can be reasonable about it. I guess it’s not really my view is so much, I don’t care one way or the other. It’s more I’m ambivalent, I don’t think it’s obvious that there’s a big error one way or the other.


Prompt: All right. Let’s talk about something pretty different which is you recently wrote a post about why divesting from companies that do harmful things could, in moderation, actually be quite an effective way to improve the world. That’s in contrast to what most people who’ve looked into that as part under the rubric of effective altruism have tended to conclude, which is that it’s actually not that useful because if you sell shares in a company or don’t lend money to a company, then someone else will just take your place, and you haven’t really made any difference.Do you want to explain the mechanism by which divesting from harmful companies, I guess like cigarette companies, for example, could be useful.

Response: Yes, I think there’s two important things to say upfront. One is that I’m mostly thinking about the ratio of costs to benefits. You can end up for some companies in a regime where divestment has relatively little effect, but is also quite cheap. In general, I think the first epsilon divestment will tend to be literally free, the cost of second-order in terms of how far you divest in the benefits or first order. That is almost always going to be worth it to divest at least by that epsilon.That’s the first part of the picture. This can be mostly a story about the costs being very, very low, rather than benefits being large, and if the costs are very, very low, then it’s mostly an issue of having to do the analysis and having to deal with the logistics in which case, I think it is plausible that one should– You can imagine really getting those costs down if someone both did the research and actually produced a fund.I could imagine myself personally being sure, I will put 0.1% of my wealth in some fund. There’s just this roughly market-neutral thing that shorts all the companies I really don’t like the activities of and goes long on most of those correlated companies. That’s one thing, it might just be about the costs and benefits both being small, such as it’s not going to be a big deal for an individual investor is maybe not worth thinking that much about it, but if someone was willing to produce a product that could be scaled a lot, and everyone could just very quickly or very easily buy the fund, then they might do that.Maybe the second thing in terms of how it could actually be possible or why it isn’t literally completely offset. I think the rough mechanism is when I get out of a company, like let’s suppose I care about oil and I divest from companies that are producing oil that increases as I divest more. The whole way that does good is by increasing the expected returns to investment in oil companies. The concern is other investors will just buy continue putting more money into oil companies until the expected returns have fallen to market returns because otherwise why not just keep putting more money in. The thing that simplified picture misses is that there is idiosyncratic risk in the oil industry namely as oil becomes a larger and larger part of my portfolio, more and more of the volatility of my portfolio is driven not by what is overall going on in the market which is the composite of many sectors, just volatility in oil in particular.If I try and go overweight, like 10% oil, like there was a lot of divestment and people had to go overweight by 10% to offset it, they would actually be significantly increasing the riskiness of marginal oil investments. The returns that they would demand in order to offset that risk would also go up. There’s two things. One is like it actually does require doing a kind of, well, it depends a little bit how rational I believe investors are. In some sense, the divestment story, like the pessimism already relied on rational investors, so it’s maybe more reasonable to say, let’s actually dig in and see how rational investors would respond and do those calculations. That’s like yes, that’s my perspective. I think it’s unusually reasonable to look into that when pessimism is coming from this home economic model.Once you’re doing that, then there are two questions. One is just qualitatively, how large are the effects that we’re talking about and it’s something I tried to run through in this blog post and a thing that I was surprised how large they were when I thought about a little bit.Maybe the second observation is that actually there’s this cancellation that occurs where roughly speaking if the oil industry, it has no idiosyncratic risk or has very low idiosyncratic risk, your divestment will get almost entirely offset but at the same time it has almost no cost to you because the industry has almost no access returns because those returns would be tied to the idiosyncratic risk.You end up with actually the fact that the cost-effectiveness doesn’t depend. This parameter which governs like how much your divestment is going to get offset, that cost-effectiveness doesn’t actually depend, or like the ratio between costs and benefits doesn’t depend on that parameter because it affects both costs and benefits equally. It affects like what is the overall upside to organizing this fund like how much will get offset but it doesn’t affect like how attractive is it for an individual investor. So like I think, like yes, we could go into the details and talk about how much it makes sense to divest. It might often make sense to divest completely or maybe even go like 100% or 200% short.In industries you don’t like, it’s going to become better, like an example of divestment that might be particularly cost-effective is like suppose there’s two companies who are producing very similar products and so are very correlated like maybe two companies that both produce poultry and one of them has substantially worse animal welfare practices.You might think there’s a lot of the risk in animal agriculture, in general, which is going to be experienced equally by those two companies. So if you’re long the company you like and short the company you dislike that has relatively little risk and it still has idiosyncratic risks specific to those companies and there’s a complicated analysis there, that like you can end up with relatively little risk compared to how much effect you have on capital availability for the two companies. We can actually talk about the mechanism by which this causes less bad stuff to happen in the world. Here we’re really just talking about why I’m skeptical of the skepticism.


Prompt: Yes. Let’s set that aside for a minute. Just to explain this in really simple language. The previous thinking has been that if you sell shares in a company then someone else who just doesn’t care about the moral issues to do with raising chickens or producing oil, they’re just going to swoop in and buy it at the same price and the share price won’t be changed or the amount of money that the company can borrow won’t really be changed. The thing that misses, is that if a decent number of people or even like a small number of people stop buying oil stocks then say, like, rich investment funds that don’t care about the moral issues, for them to go and buy even more of these fossil fuel companies that they don’t want to do that or their willingness to do that isn’t unlimited because they want to be diversified across all of the different assets in the world. In order to buy extra oil shares to make up for the fact that you and I don’t want to own them, they have to reduce the diversification that they have which is unappealing to them.If a bunch of people or even just a few short or sell these shares, it actually probably will suppress their price a little bit because people will have to be compensated for the reduced diversification with a lower share price to make it more appealing to buy. That’s one thing. Also, while that effect might be pretty small in the scheme of things, it’s also the case that just like selling those first few shares of those oil companies, it wasn’t important to you to own those specific companies anyway. It’s just like you can like slightly reduce your diversification. Just sell tiny amounts of these companies that you own in your portfolio, which costs you practically nothing. Even though the benefit is quite small, the costs could potentially be even smaller because this just doesn’t matter that much. Then the ratio of benefits to costs could be pretty large even if like this is not the best way to have an impact in the world.

Response: Yes. That’s right. If you want to think about what the total impact is like, it’s reasonable to imagine scaling this up to large numbers of investors doing it like a lot of the effects are going to be roughly linear in the relevant range. The total impacts are not that bad. They don’t look great but they look like if you imagine a change where everyone like large factions of people divested I think it would meaningfully decrease the amount of oil that gets extracted or the number of chickens raised in captivity, especially in cases where you have– I think maybe the oil case is a little bit unfavorable in this way compared to the chicken case where you could really imagine slightly shifting towards more practices of better animal welfare. Yes. You can imagine slightly shifting towards practices that are better for animal welfare or towards different kinds of meat or so on.The total effect is probably not that big. The total effect may still be large enough to justify really getting the logistics sorted out. It’s very easy to do. It’s from the investor’s perspective other than the hassle of doing it, I think it’s actually pretty definitely the first unit is a very good deal.


Prompt: Well, couldn’t you just buy into an investment fund that doesn’t own oil companies or doesn’t own animal agricultural companies? That seems like the first part, that’s pretty straightforward to do.

Response: Yes. Also, I’m thinking then, so when I buy a fund there’s a bunch of things that constrain my choices and it’s kind of annoying if now I have this extra constraint on top of those that might be a reason and–


Prompt: It’s not quite worth it?

Response: Yes. Even if you slightly raise management fees on that fund. So like Vanguard is going to offer me some tinsy–


Response: Yes. Now I have to pay 0.1% on the same thing. That’s no good. I would normally imagine my baseline implementation would be a fund that shorts the relevant, the particular companies you care about and maybe also opens up the offsetting rates. The reason it was bad to sell this company is because we were losing diversification so they can try to do things to offset those costs as part of the same bundle. I would be very interested just seeing that there’s the optimal divestment fund for people who care about animal welfare or whatever. That just holds mostly if there’s really large short positions in the company that have the worst animal welfare effects.Then also construct a portfolio to capture as much as possible the diversification benefits that those would have added to your portfolio. The cost of investing in that can be pretty low and you can just then put that on top of– do whatever else you would have done in investing, then take 0.1% of your money or whatever, 1% of your money and put it in this fund. On average that fund is going to make zero dollars and it’s going to have some risks. The cost to you is just the risk of this fund, that on average, is making no money but it could be relatively- if you’re not investing that much of your money in it, the risk is just not that bad.


Prompt: To what extent, if it is, is this analogous to being useful to not go and work at an evil company?

Response: I think it is fairly analogous. There’s a bunch of quantitative parameters. If you take a certain economic perspective they’re very structurally analogous. The discussion we’re having about risk, which is important to determine some of the relevant elasticities is quite different from the analogous discussion in the case of working in a problematic industry. I think the overall thing is kind of similar. Well, if you don’t work in that industry overall what happens is that prices go up or wages go up a little bit in the industry and that induces more people to enter. We just have to talk about how much do wages go up.One thing I sort of think about is if we consider say animal agriculture. It’s also kind of analogous to the discussion with ethical consumption. I think that’s actually a really good comparison point for divestments. Where you could say, “I want to consume fewer animal products in order to decrease the number of animals we get produced.” Then you have a very similar discussion about what are the relative elasticities are like. One way you could think about it is if you decrease demand by 1%, you decrease labor force by 1% and you decrease the availability of capital by 1%. If you did all of those things then you would kind of decrease the total amount produced by 1% roughly under some assumptions about how natural resources work and so on.The credit for that 1% decrease is somehow divided up across the various factors on the supply side and demand side and the elasticities determine how it is divided up. I think it’s not like 100% consumption or like 100% of labor, I think all of those factors are participating to a nontrivial extent.The comparison to ethical consumption I think looks reasonably good. I think under pretty plausible assumptions you’re getting more bang for your buck from divesting from– I haven’t done this analysis really carefully and I think it would be a really interesting thing to do and would be a good motivation if I wanted to put together the animal welfare divestment fund. I think under pretty plausible assumptions you’re getting a lot more bang for your buck from the divestment than from the consumption choices. Probably you’d still want like the consumption– the investment thing would be relatively small compared to your total consumption pattern. It wouldn’t be like replacing your ethical consumption choice. If ethical consumption was a good idea, then also like at least totally divesting and maybe even 10X leveraged short positions. When you would have bought one dollar of animal agricultural companies, instead you sell $10. I think stuff like that could be justified if you thought that ethical consumption was a good thing.


Prompt: Do you just want to map out, or sketch out briefly for those who are skeptical, how it is that selling shares in a company or selling bonds in a company reduces the output of that company?

Response: Yes. I think the bond case is a little bit simple to think about though I probably think they’re probably about the same. Let’s talk about the bond case. This is the company Tyson wants to raise a dollar. They’ll go out to investors and say, “Give us a dollar now and we’ll give you some amount of money 10 years from now assuming we’re still solvent.” That’s their pitch. They’re selling these pieces of paper to people which are like ‘I owe yous’. The price of those ‘I owe yous’ or how much the ‘I owes yous’ have to be forged are set by supply and demand amongst investors.What happens when you short? The bond is– someone came to this company and wanted to loan them a dollar and they’re saying, “Don’t loan them a dollar. Instead, loan me a dollar, and whatever it is that they pay back to their bondholders, I’ll pay it back to you instead.” They’re like, “Fine. I’m just as happy to lend to you as I was to lend to the actual company.” Now the company has one less dollar. Now the company’s like, “We still need to raise that dollar if we want to produce this additional marginal check-in.” Now the company goes and tries to raise the dollar but they’ve used up one of the willing buyers. Now they need to find another buyer, someone who’s willing to loan them this dollar.That person is going to be a little bit less excited because, again, their portfolio is a little bit more overweight in this company, so they’re a little bit more scared about the risk of this company going under. Roughly speaking, that’s the mechanism.


Prompt: I think it makes sense. You imagine what, if just a lot of people weren’t willing to lend money to a company with significant number and this drives up their borrowing cost, and so the company shrinks because they have to pay a higher interest rate. They can’t get as much capital. It kind of makes sense on an intuitive level. Some of this gets a little bit technical, so we’ll stick up a link to the blog post that you write with all the equations, explaining how you worked this through and try to estimate the size of the benefits and the costs.

Response: I’m concerned it’s not the most careful or clear analysis to people. I think I’m interested and I think at some point, will have a more careful version that I put up. Just a fun exercise for me.


Prompt: You make some points that I haven’t seen anywhere else, and that actually might shift the conclusion. That seems like probably the most important thing that people need to take onboard.

Response: That’d be super interesting to me if I actually ended up with the divestment fund that was reasonably the long-short fund that was recently constructed and cost-effective. That would be kind of cool. Also, sorry. I said they’re like compared with ethical consumption. I think one thing I want to stress there, is that the way that work is getting done is just because of the slight changes on the margin being very effective. It’s very similar to vegetarians. If you just stop eating meat in cases where it was really marginal, that has a lot more bang for your buck than if you go all the way.It’s the same thing here. It’s not going to be competitive with the first unit of stopping eating meat. It’s going to be competitive with going all the way to the last bits.


Prompt: It’s going to be a little bit embarrassing if effective altruist aligned folks have been saying divestment is a waste of time for all these years, and it turns out that we’re pretty wrong about that. [laughs] Going to have to eat humble pie. I suppose it also looks good that we’re updating our views so we’re not just stuck with dogmatic positions.

Response: I think we also most likely end up with some compromise where we’re like, look, the impacts are a lot smaller than people will often implicitly assume when they are pitching this. It is, on balance, like a reasonable thing to do. Maybe we shouldn’t have been quite so down on it.


Prompt: The costs are negligible a lot of the time.

Response: Or it is really a social thing of the cost of just people believing. The only difficulty to make it happen is people believing they should it. Therefore, it’s like a reasonable– If there’s a change people can make that costs them almost nothing, it’s a particularly reasonable thing to advocate for people to do.


Prompt: Let’s talk about s-risks for a minute. Some listeners will know, but some people won’t, that s-risks is this term people have settled on to describe possible future scenarios that are just neutral where humans go extinct and then there’s nothing, or it’s not very good, where humans stick around but then we just don’t make the world as good as it could be, but rather worlds where there’s astronomical levels of bad things. I guess S, in this case, stands for suffering because a lot of people tend to be concerned that their future might contain a lot of suffering.It could also just include any future that is large in the sense that a lot of stuff is going on, but it also contains a lot of bad stuff rather than good stuff. Some of the ways that people are aware this could happen involved in artificial intelligence that doesn’t share our goals. What’s your overall take on s-risk as a problem to work on?

Response: I think my best guess is that if you go out into a universe and optimize everything as being good. The total level of goodness delivered is commensurate with the total amount of badness that will be delivered if you went out into the universe and optimize it for things being bad. I think to the extent that one has that empirical view or like that maybe moral, some combination of empirical view and moral view valid the nature of what is good and what is bad, then S-risks are not particularly concerning because people are so much more likely to be optimizing the universe for good.So much more in expectation, the stuff in the universe is optimized for exactly what Paul wants rather than exactly what Paul doesn’t want. That’s my best guess view. On my best guess view, I think this is not a big concern. I think I do have considerable moral uncertainty. I guess the way that I would approach moral uncertainty, in general, would say that even if an expectation, it’s sort of hard to talk about it in comparing the expectations of outcomes across these very different moral views. This is one of the cases that gets the comparison that’s difficult because of those weird difficulties with inter-theoretic utility comparisons.The way I would normally think about this kind of case is to say I should put reasonable priority or reasonable interest in reducing S-risks, if I put a reasonable probability on views on which the total amount of possible badness is much larger than the total amount of possible goodness which is where I’m at.I think it’s not likely, but plausible combinations of empirical and moral views on which they’re very important. That’s my starting point of taking this as a thing which I’m not going to put much on because I don’t find that perspective typically appealing when it’s not going to be a large fraction of my total concern. It deserves some concern because it’s a plausible perspective.


Prompt: The naive take on this might be, “Why would we worry about these scenarios because they seem really outlandish? Why would anyone set out to fill the universe with things that are really bad? That just seems like a very odd thing to do.” Once you’re at the level of sophistication where, yes, you can go out and colonize space and great astronomical massive stuff, why are you filling it with stuff that’s bad.Then there’s something to be said for that but then people try to think about scenarios in which this might happen, which might involve conflicts between different groups were one of them threatens the other that they’re going to do something bad and then they follow through in doing that, or potentially where we don’t realize that we’re creating something that’s bad.You might create something that has a lot of good in it, but also has a bunch of bad in it as well and you go out and spread that. We just don’t realize that we’re also as a side effect to creating a bunch of suffering or some other disvalue. How plausible do you think any of these scenarios are?

Response: I guess the one that seems by far most plausible to me is this conflicts, threats and following through on threats model.


Prompt: Not just moral error potentially?

Response: I think it’s hard to make a sufficiently extreme moral error. There might be moral error that’s combined with threats they could follow through on but I think it’s hard for me to get- the risk from that is larger than the risk for me getting what is good, very nearly exactly backwards. It’s not totally impossible to get things exactly backwards. It’s more likely than hitting some random points in the space for a wide variety of reasons but I think it’s still a minority of my total concern. Most of it comes from someone wanting to just destroy shit because they wanted to have a threat of destruction or destroying value, so that’s what I would mostly be worried about.


Prompt: How plausible is it? It seems like you think it’s conceivable but pretty unlikely, so it’s like you pay a little bit of attention to it, but it’s not going to be a big focus. Is that kind of the bottom line?

Response: Yes, so when I was talking before about these comparisons, how being conceivable means it gets a little bit of priority. That was more with respect to sort of moral views or aggregation across different values and how much weight I give them?Well, I think the key question is how much credence do you place on views where the worst outcomes are much more bad than the best outcomes are good and then I think that those views basically are going to recommend if the ratio was large enough, just focusing entirely on minimizing this risk of really bad stuff. I think regardless of one’s empirical view, it’s worth to put in some amount of attention to reducing the risk of really bad stuff.In terms of how plausible it is, that’s still important to understand the basic shape of what’s up. I don’t really have a conservative view on this. I think the answer is, it’s relatively unlikely to have significant amounts of disvalue created in this way but not unlikely, like the one in a million level, unlikely, more like the 1% level.As a question of when they’re bad, what fraction of the badness is realized compared to the worst possible outcome and how much of the universe’s resources go into that, and that estimate is not very stable. That’s where I’m at.


Prompt: You made this argument at the start that it seems naive that you would think that it’s as easy to create good things, as to create something that’s equivalently bad and so more future beings are going to want to create good things and bad things, so we should expect the future to be positive. How confident are you that actually is true, that it’s symmetrically easy to create good and bad things?

Response: When you say symmetrically to create good and bad things, I think it’s worth distinguishing, being clear about what exactly that means. I think the relevant thing here, assuming that we’re linear, things twice as big or twice as good or bad, then the relevant question is just what is your trade-off?Suppose you have a P probability of the best thing you can do and a one-minus P probably the worst thing you can do, what does P have to be so it’s the difference between that and the barren universe. I think most of my probability is distributed between you would need somewhere between 50% and 99% chance of good things and then put some probability or some credence on views where that number is a quadrillion times larger or something in which case it’s definitely going to dominate. A quadrillion is probably too big a number, but very big numbers. Numbers easily large enough to swamp the actual probabilities involved, a quadrillion is just way too big. I should have gone with ‘a bajillion’, which was my first…Anyway, in terms of how confident I am on the 50% or on the 50% to 99%, I think I would put a half probability or like weight a half to a third on the exactly 50 or things very close to 50% and then most of the rest gets split between somewhat more than 50% rather than radically more than 50%.I think that those arguments are a little bit complicated, how do you get at these? I think to clarify the basic position, the reason that you end up concluding it’s worse is just like conceal your intuition about how bad the worst thing that can happen to a person is the best thing or damn, the worst thing seems pretty bad and then the like first-pass responses, sort of have this debunking understanding, or we understand causally how it is that we ended up with this kind of preference with respect to really bad stuff versus really good stuff.If you look at what happens over evolutionary history. What is the range of things that can happen to an organism and how should an organism be trading off like best possible versus worst possible outcomes. Then you end up into well, to what extent is that a debunking explanation that explains why humans in terms of their capacity to experience joy and suffering are unbiased but the reality is still biased versus to what extent is this then fundamentally reflected in our preferences about good and bad things. I think it’s just a really hard set of questions. I could easily imagine maybe shifting on them with much more deliberation.


Prompt: Yes. How do you think technical AI research or your focus would change if preventing S-risks became a high priority?

Response: I think the biggest thing is this understanding better the dynamics that could possibly lead to bad threats being carried through on and understanding how we can arrange things. It’s less likely for that to happen. I think that’s the natural top priority.


Prompt: I heard an interesting suggestion for how to do that recently which was– Concern you might have is that someone would threaten to create the thing that you think is really disvaluable. Let’s say I’m concerned about, I don’t want suffering to exist in the future. That leaves me open to someone threatening to create a suffering in order to get me to concede on some other point. But I could potentially avoid that risk by say, changing myself so that I also disvalue something that was actually not important at all. Let’s say I want to- I also really don’t like there being– I don’t want there to be like flying horses or something like that. Something that doesn’t exist.In that case, if someone wanted to extort me there or wanted to threaten me then they can instead rather than threaten to create suffering, they would instead have the option of threatening to create flying horses which currently don’t exist but they could threaten to create them. Potentially, I could change my value such that it’s more efficient to create that than it would be to create suffering and so that would be the most efficient threat I think to threaten me with. It’s kind of this spillover like part of your utility function that protects you from threats about the things that you previously cared about. Do you have any reaction to that idea or things in that vein?

Response: I think my initial take seemed crazy and then since then I have become significantly more enthusiastic about it or it seems plausible. I think actually one of the like I was giving out a prize last year for things that seemed relevant to AI alignment or to AI leading to a good outcome. I think one of them, Caspar from EAF, gave some version of this proposal. He submitted some proposals along these lines and at that point I thought about it more and I was somewhat compelled. I think since then he’s continued to think about that and that seems interesting.I think a perspective on that that I find somewhat more plausible than like don’t care about that thing is, you could say I care a lot about this random thing like how many flying horses there are. You could also take this perspective that’s kind of like a big bounty. If you were to demonstrate to me convincingly to you could have run this strategy, it would have had a significant chance of causing extreme disvalue and would have coerced me into doing X. It would have in fact caused me to do X. You can just demonstrate that sufficiently convincingly and then really once you’ve persuaded me of that I’m like, “Hey, fine.” You can have whatever outcome you would have in fact achieved, an outcome which is from your perspective like incrementally better than whatever outcome that you would have achieved by carrying through this risky policy.It’s not clear. I think it’s incredibly complicated. I’ve started to spend a little bit of time thinking about this and it’s just incredibly complicated to figure out if it’s a good idea. Well, not if it’s a good idea but whether it really works. I think it’s like a thing I’ll be interested in people thinking more about it. It’s definitely one of the things I’ll be doing and understand the conditions under which bad threats could fall through on. I think that makes less difference than other common-sense cool interventions like avoiding the situation where there are people threatening each other. It is a lot easier to make, to get into flexible traction on these more obvious open-questions there.


Prompt: One reason that people work on S-risks is that they are more worried about preventing bad things than they are about creating good things. Another rationale might be, even if you are symmetric in that point I would be that there’s more people working on trying to prevent extinction or trying to make the future go well than there are people worrying about the worst-case scenarios and trying to prevent them, so it’s like potentially right now a more neglected problem that deserves more attention than it’s getting. Did you put much, or any weight on that?

Response: I think ultimately I mostly care about neglectedness because of how it translates to tractability. I don’t think this problem is currently more tractable than– I don’t feel like it’s more tractable than AI alignment. Maybe they seem like they’re in the same ballpark in terms of tractability. I don’t think, part of it is like a harder problem to deal with. It also have these concerns where it’s not– There are a bunch of reasons, it’s like maybe less tractable on its face than alignment.


Prompt: Why is that?

Response: I think the basic source of a lot of difficulty is that part of the source is the threat model for alignment is incredibly clear like you have this nice model in which you can work. You understand what might go wrong. I mean it’s absurd to be comparing alignment to a problem and be like it’s incredibly clear and concrete. That basically never happens. Anyway, but in this one comparison, we can be like it’s unusually much more clear and concrete whereas here we’re like, “Geez.” It’s quite a fuzzy kind of difficulty and the things that we’re going to do are all much more like bing shots. I don’t know, I think it’s a messy subject.


Prompt: Quite a lot of people think that these risks of bad outcomes and threats are more likely in a multipolar scenario where you have a lot of groups that are competing over having influence over the future and I guess over potentially the use of artificial intelligence or whatever other technologies end up mattering. Do you share that intuition?

Response: I think it’s at least somewhat worse. I don’t know how much worse like maybe twice as bad seems like a plausible first pass guess. The thing is turning a lot on how sensitive people are threatening each other in the world. That seems bad. That’s like one major source of threats and it’s like if you have fewer, if you have less rapid competition amongst people you’d expect to have less of that going down. Some questions about how sensitive the number of threats people are making against each other to amounts of polarity seems pretty sensitive. Then what fraction of all threats occur say over the next hundred years and that kind of dynamic.


Prompt: Do you have any thoughts on coordination between people who are mostly focused on making the future contain good things and people who are mostly focused on making sure that it doesn’t have bad things?

Response: Mostly, I think the reason they’re going to end up coordinating is via they’re pursuing similar approaches like cognitive styles for thinking about the situation and people should be coordinating. Generically, it’s nice if people can coordinate even if their goals are on the object level– Even if you have totally orthogonal goals, it’d be good if you share resources and talk and benefit from the resources of each other and so on. We do have somewhat- people normally don’t have extreme bias of one end to the other. That’s the main channel for coordination, you’d also hope for some kind of more cooperation, just through overlapping objectives that happen to serve both. I think that’s like a less important channel here like both communities can be happier and healthier and get along better if we all care to some extent about these different things and we, therefore, should all help each other out.


Prompt: Let’s talk a second about philosophy and ethics and AI. What role do you think different theories of metaethics play in AI alignment and potentially AI alignment research?

Response: I think there are two qualitatively different ways that philosophical progress could affect AI alignment. One is on the object level thinking that the work we need to do when aligning an AI involves clarifying some philosophical questions. A different one is like the way that you’ll approach alignment depending on your views on some of those philosophical questions. I think it’s worth distinguishing those.So on the object level, if you thought that you had to understand some process which would ultimately converge to a correct understanding of the good and that you had to directly impart that into an AI system you built, then it’d be a really rough position where you either have to solve a bunch of philosophy yourself, what Wei Dai calls metaphilosophy, understanding by what process humans arrive the truth when doing philosophical inquiry. That seems pretty rough.Then there’d be this really tiny object-level connection where you might even end up saying these are basically the same problem. I think that’s a perspective maybe that’s closer to where I was six years ago and I’ve really shifted a lot towards like, “Look, you just want to have a system that is– The thing you want to be clarifying is this notion of control and course-correction.” You want to say we want the construction of the AI to not make anything worse. We want to end up in a position like the one we’re currently in where we get to continue going through the same process of deliberation and understanding the ways in which that process goes well or poorly and correcting them and avoid– We want to be making essentially as few ethical commitments as we can at the point we’re constructing AI. And I’ve become much more pessimistic about any approach that essentially involves any hard philosophical commitments. I think we’re still end up making some and I’m happy to talk about the ones that we most likely to make but I don’t think– I think if things go okay, it’s probably because we can dodge most of them.


Prompt: Why have you become more pessimistic?

Response: I think in part it’s from technical details of the case and just thinking about how different approaches to alignment might play out. By that I mean something like I think you have to be leaning on this mechanism of course-correction by humans or like deferring to this process of human deliberation a lot anyway to have any hope apart from philosophical issues and once you’re leaning on it, in general, you might as well also just lean on it for answering these questions. In part, it’s just becoming a lot more optimistic about the prospects for that. You might ask how important is it that I understand what you want to happen with the universe as a whole when it goes out and acts on your behalf. I’ve opted a lot towards it’s okay if it doesn’t really understand– Even really pessimistic cases where stuff is getting crazy.There’s going to be like six minutes between when you start your AI and when the colonization of the universe begins. I think even then it’s like basically, if it doesn’t understand what you want for the universe that much. Just understands like, “Look, here’s me.” Put you in a box somewhere. Now, let’s start colonizing the universe and then eventually make some space for your box to sort out what humanity should do like a little civilization. I plan it somewhere out in the backwoods trying to figure out what is good and then ultimately the process just remain responsive to the conclusions of that deliberation.The thing that has to understand what does it mean to protect humanity and allow humanity to develop and mature in the way that we want to develop and mature. Then what does it mean to ultimately be responsive to what that process concludes? To be correctable, once humans figure out what it is we want in some domain. Allow that understanding to ultimately affect the behavior of this scaffolding of automation we’ve built up around us. There’s maybe one last more technical question that comes up there. You might think philosophy would affect the value of different kinds of resources and there’s some more– I think you can sort of dodge these kinds of dependence once you’re more careful about these arguments


Prompt: How do you feel about the idea of the long reflection of this idea that we don’t really know what’s valuable. It seems like we might have a better shot if we get our best people to think about it like thousands of years or just a very long time until we decide what would be the best thing to do with all of the resources that we can get in the universe. Is that a sound idea?

Response: I think viewing it as a step which occurs is probably not quite right. I think I’m pretty on board with the idea that is like the process of deliberation, understanding what is good and we’re currently engaged in. I see most of the action in AI is allowing that process to continue happening.I think it can be that processes like to couple ultimately from most of the economic expansion through the universe which is decoupled from this process of ongoing deliberation, understanding what we want. You can imagine a world– where humans are living on earth while out there in space, a bunch of crazy shit is going down.AI is waging wars and building machines and stuff, and the humans are just like we’re doing our normal thing on earth. Sometimes we need futurists to more think of that as a sideshow where all the action was now off with the crazy stuff the AIs are doing.The effect was more shifted to– Actually, that’s where a lot of the action is. The overall evolution of our values. We choose what we want to be the store of value. I don’t think it’s like a person going off and thinking for a thousand years it’s more like there’s a trajectory along which our civilization is developing, and we’re thinking about how that trajectory should be.I think that one of the things that happens– One of the hopes of AI alignment is to decouple that process of ongoing deliberation from the process of remaining economically competitive. It’s a really hard problem to understand what that deliberation should look like.That’s like another one of the big ways. When I said at the beginning, that I’d slightly downgraded my overall sense of how important alignment was to other parts of the problem. A lot of that has been up -weighting how important making natural deliberation go well is .That’s really from a long term perspective other than alignment that seems like probably top priority. I think that’s also worth– At the beginning, we talked about metaethics, I made this distinction between object level and meta-level influences.It’s worth bracketing . Here we’ve just been diving in on the object level. I’m like happy to keep going with that. It’s worth saying briefly that at the meta level that like, I do think that your approach to alignment about like, how important, how valuable different kinds of outcomes are, depends on the answers to some hard ethical questions related to this like how much do you care about ” the lizard people”?That’s a hard question in moral philosophy and similar hard questions are how much do you care about this AI system that you’re building? If you want to do something totally different from humans, how much should you be like, well, we have some values AI some values, we are happy.I think we talked about this a little bit on the last podcast. The answers to those kinds of moral questions does have an effect on how you go about alignment or how you prioritize different aspects of the problem.


Prompt: What are your views on moral realism and anti-realism? Do those affect what AI alignment work seems most important?

Response: I’m definitely pretty anti-realist. I think we get a little bit into semantic weeds here when I’ve had long discussions with people about it. I think there’s this question which feels like a realist question of, “Is deliberation going in the right direction?”I don’t think you could have a version of the anti-realist perspective where you’re like it doesn’t matter how you deliberate, you’re going to come to some conclusions and those are fine. I don’t endorse that. There’s another version of this perspective where you shouldn’t deliberate, you should just endorse your current conclusions because that’s what you like.I don’t endorse that either. I’d say like look, right now there’s some kind of processes in deliberation and growth, but I endorse the output of this. In some way I want our values to evolve. In some sense, you could say that what I care about is the endpoint of that deliberative process.The endpoint of that potentially very long process of evolution maturation. I think I philosophically don’t think there would necessarily be convergence across different– I think different process and emotion would arrive at different conclusions.I think there is a very hard problem of having an echo in the right direction. It’s a little bit awkward as a non-realist to be like what is the right direction mean here? The realist has this nice, easy answer like is actually converging to the good.I think that’s just a linguistic thing where they happen to have a nice– again, the whole thing is semantic differences. It’s just like there’s some concepts that are slippery or hard to talk about and think about for the non-realist.I think that’s because they are, in fact, for the realist, let’s just push down to the slipperiness and complexity of the actual concept of good. In terms of what I find … my view overall on the object level questions that might affect how to prioritize different parts of alignment is I don’t think there’s that much convergence.I think it’s quite plausible that an AI would be smart and would do its own thing. That’s somewhere between a barren universe and actually achieving the optimal outcome and I don’t think it’s very close to barren. I don’t think it’s very close to the optimal outcome.I lean towards like a 50/50 prior for like I’d be okay… I’d treat it half as bad as extinction. Like, to have some random AI doing this random thing with the universe.


Prompt: We’ve been here for a while and we should wrap up. Earlier we were talking about how creatine could potentially make people a whole lot smarter, or sorry, three IQ key points smarter. [laughs] A third of a standard deviation it’s as good as it gets potentially.Do you have any view on nootropics and kind of the drugs and life-hacking stuff that people try to use to make themselves fitter and more intelligent? Is there much mileage to be gotten out of that?

Response: I’m definitely pretty excited about investigating where it feels like a thing that the medical establishment is not really not into but like very not into. I think there’s a reasonable chance that there’s something I think there are a few cases where the current state of the literature would — If you took everything at face value, you’d be like it does seem like I don’t know. I think maybe Piracetam is in this position, but I’m not totally sure. There’s a few other possible candidates. If you actually believed all the studies and just took that at face value, you would think that there’s reasonable gains from some of these.I think probably that’s just not the case. Everyone understands that these are older studies and there have probably been failed replications that haven’t been published. It would be pretty nice to go through and check them all and I’d be pretty excited about that.I’m pretty interested in general– I think the thing I would most like to see– I generally have this question of why would it be the case that some simple changes to brain chemistry would improve performance but not be made by evolution. You want to see what the countervailing consideration is.


Prompt: That does potentially explain why it has been pretty hard to find anything that seems like it works really well. If it was as easy as that then evolution would have done it.

Response: Yes, so I think you’re going to have to exploit some distributional change or some cost which is now a cost that wasn’t a cost historically. I think the two best candidates are basically, one, you can exploit this hypocrisy angle. People have this thing they want to do like, “I want to make the world better,” but at some level, it looks like your biology is not at all optimized for making the world better. It’s optimized for having descendants from descendants. One thing is if you want to pursue what you nominally want to do and you can hope that there are some drugs that just make you better at achieving the thing that you set out to do. Even when that thing is not in line with the thing that your body is optimized for achieving. I think that’s a mechanism of action in some cases and seems pretty realistic. It’s something I’ve been really scared of.Then the other one, the thing that would be most satisfying and excellent would be it just burns more energy. In an evolutionary environment, you didn’t want to run your brain hot because it’s kind of a waste and you’re only getting marginal benefits but if you just could do that, that would be super great.


Prompt: Just overclock it because it’s like, “We’ve got so much food now.”

Response: So much food. [laughs]


Prompt: You told me I’m going to burn an extra 400 calories a day and be marginally smarter, I’m like, “That’s good on both sides.”

Response: Yes. [laughs] That would really be the best case definitely. I’m a little bit confused about where stimulants stand on that. Usually, my understanding for caffeine is that if I take caffeine I’m probably going to drive up blood pressure and drive up energy expenditure, at least at the beginning. Then if I keep taking it, probably within two weeks, at least on the blood pressure side, I’m going to return to baseline. I would like to understand better what long-term effects are on cognitive performance and energy use.I’d similarly like to understand whether those long-run effects. Is it the case that if you take something over and over again then eventually it stops having impact. Or is it the case that if you take it even once you have the bounce-back period over the next couple days such that the long-term effect is the integral of this thing that are showing that the integral is zero. I don’t know, anyway, it seems plausible to me there’s some wins, probably mostly through those two channels of either getting it to do what you think you should do, or trying to do, or else bring in more energy.I have personally have not prioritized experimenting with that kind of thing. In part, because I am really bad at introspection, so I cannot tell even if I’m in a very altered mental state. Partly because I think it’s really hard to get results with n=1, but I’m pretty excited about more experimentation on those things. It’s really hard probably because the medical establishment really seems to hate this kind of thing. Some of the likely winners are also not legal which sucks. Like, prima facie, amphetamines would probably be the most natural candidate.[laughter]


Prompt: Do you think they would be good in the long-run? It does seem like the body is so good at adapting to stimulants that most of the time they work for the first week or the first month but then you’re back to baseline and now you’re just taking this thing that just returns you to normal.And while almost all of these cognitive enhancers are legal, with a handful of them like amphetamines, there’s a risk of getting into legal trouble if you take them, or even not being able to get a security clearance in the future if you’ve taken them, which is a really serious downside for anyone who might go into policy careers in future. And people who might really want or need a security clearance at some point make up a large fraction of all our listeners.So in those cases, it’s clear you should stay away.But, basically, I don’t recommend that people pay much attention to nootropics, both because of the fact that for many the evidence they work is weak in the first place, combined with the fact that the body is so good at adapting to and undoing the effect of almost anything you take after a few weeks.

Response: That’s a lot of my concern. I mean you could imagine. I think people have a sense that like under– Some people are prescribed stimulants for example, have the sense that under-repeated use, you do get some lasting advantage or at least lasting treatment effect. It would be good to check whether that’s actually the case or whether that’s shuffling things around and zero-sum. It does kind of feel like once you’re burning more energy, there’s not really a good reason. It’s like, if I’m trying to take some Piracetam and hope that I think better. There’s sort of a good reason to expect to fail and then lose to this evolutionary argument.In the case of a stimulant which is initially causing you to use more energy and think better, there’s not really a great reason to expect it to break down. So there’s hope at least. I think in my ideal world you’d really be throwing a lot of energy like, “Can we find a win here because it sure seems plausible.”


Prompt: One hope I’ve had is that even if the brain is– It seems like if you take a stimulant and then you get a benefit from it but your body is adapting to it and then you have to have a time when your body de-adapts from it in order to take it again. If you say take it all the time for a week and then don’t take it for a week in order to flush out the adaptation from your system, that doesn’t seem so great. Potentially, you could do this every day, so that you’re a bit more awake during the day and then you de-adapt to it while you’re sleeping and that seems potentially good on both sides. I’m not sure whether the de-adaptation over the 12 hours that you’re not taking it is really sufficient to make it worthwhile.

Response: Yes, I think it feels plausible to me there’s something here that works. It’s plausible to me that you should just run with something dumb like, “I thought about it a little bit on paper, I tried some.” I think it’s hard to do the experiments due to the combination of that the medical establishment hates it, the instruments are a little bit of the hard thing, you’d have to get over that.Also, the most high-leverage options probably are not going to be legal, which makes it just overall less appealing. You can try this with caffeine but it seems just probably less wins.


Prompt: Well, on that inspiring note, my guest today has been Paul Christiano. Thanks for coming back on the podcast, Paul.

Response: Yeah. Thanks for having me.


Prompt: Hello everybody. Today, I’ll be speaking with Paul Christiano. Paul is a researcher at the Alignment Research Center, where he works on developing means to align future machine learning systems with human interests. After graduating from a PhD in learning theory in 2017, he went onto research AI alignment at OpenAI, eventually running their language model alignment team. He’s also a research associate at the Future of Humanity Institute in Oxford, a board member at the research non-profit Ought, a technical advisor for Open Philanthropy, and the co-founder of the Summer Program on Applied Rationality and Cognition, a high school math camp. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. Paul, welcome to AXRP.

Response: Thanks for having me on, looking forward to talking.


Prompt: All right. So, the first topic I want to talk about is this idea that AI might pose some kind of existential threat or an existential risk, and there’s this common definition of existential risk, which is a risk of something happening that would incapacitate humanity and limit its possibilities for development, incredibly drastically in a way comparable to human extinction, such as human extinction. Is that roughly the definition you use?

Response: Yeah. I think I don’t necessarily have a bright line around giant or drastic drops versus moderate drops. I often think in terms of the expected fraction of humanity’s potential that is lost. But yeah, that’s basically what I think of it. Anything that could cause us not to fulfill some large chunk of our potential. I think of AI in particular, a failure to align AI maybe makes the future, in my guess 10% or 20% worse, or something like that, in expectation. And that makes it one of the worst things. I mean, not the worst, that’s a minority of all of our failure to fall short of our potential, but it’s a lot of failure to fall short of our potential. You can’t have that many 20% hits before you’re down to no potential left.


Prompt: Yeah. When you say a 10% or 20% hit to human potential in expectation, do you mean if we definitely failed to align AI or do you mean we may or may not fail to align AI and overall that uncertainty equates to a 20%, or 10% to 20% hit?

Response: Yeah, that’s unconditionally. So I think if you told me we definitely mess up alignment maximally then I’m more like, oh, now we are looking at a pretty big, close to 100% drop. I wouldn’t go all the way to 100. It’s not literally as bad probably as a barren earth, but it’s pretty bad.


Prompt: Okay. Yeah. Supposing AI goes poorly or there’s some kind of existential risk posed by some kind of, I guess really bad AI, what do you imagine that looking like?

Response: Yeah. So I guess, I think most often about alignment, although I do think there are other ways that you could imagine AI going poorly.


Prompt: Okay. And what’s alignment?

Response: Yeah. So by alignment, I mean - I guess a little bit more specifically, we could say intent alignment - I mean the property that your AI is trying to do what you want it to do. So we’re building these AI systems. We imagine that they’re going to help us. They’re going to do all the things humans currently do for each other. They’re going to help us build things. They’re going to help us solve problems. A system is intent aligned if it’s trying to do what we want it to do. And it’s misaligned if it’s not trying to do what we want it to do. So a stereotypical bad case is you have some AI system that is sort of working at cross purposes to humanity. Maybe it wants to ensure that in the long run there are a lot of paperclips, and humanity wants human flourishing. And so the future is then some compromise between paperclips and human flourishing. And if you imagine that you have AI systems a lot more competent than humans that compromise may not be very favorable to humans. And then you might be basically all paperclips.


Prompt: Okay. So this is some world where you have an AI system, and the thing it’s trying to do is not what humans want it to do. And then not only is it a typical bad employee or something, it seems you think that it somehow takes over a bunch of stuff or gains some other power. How are you imagining it being much, much worse than having a really bad employee today?

Response: I think that the bad employee metaphor is not that bad. And maybe this is a place I part ways from some people who work on alignment. And the biggest difference is that you can imagine heading for a world where virtually all of the important cognitive work is done by machines. So it’s not as if you have one bad employee; it’s as if for every flesh and blood human there were 10 bad employees.


Prompt: Okay.

Response: And if you imagine a society in which almost all of the work is being done by these inhuman systems who want something that’s significantly at cross purposes, it’s possible to have social arrangements in which their desires are thwarted, but you’ve kind of set up a really bad position. And I think the best guess would be that what happens will not be what the humans want to happen, but what the systems who greatly outnumber us want to happen.


Prompt: Okay. So we delegate a bunch of cognitive work to these AI systems, and they’re not doing what we want. And I guess you further think it’s going to be hard to un-delegate that work. Why do you think it will be hard to un-delegate that work?

Response: I think there’s basically two problems. So one is, if you’re not delegating to your AI then what are you delegating to? So if delegating to AI is a really efficient way to get things done and there’s no other comparably efficient way to get things done, then it’s not really clear, right? There might be some general concern about the way in which AI systems are affecting the world, but it’s not really clear that people have a nice way to opt out. And that might be a very hard coordination problem. That’s one problem. The second problem is just, you may be unsure about whether things are going well or going poorly. If you imagine again, this world where it’s like there’s 10 billion humans and 100 billion human-level AI systems or something like that: if one day it’s like, oh, actually that was going really poorly that may not look like employees have embezzled a little money, it may instead look like they grabbed the machinery by which you could have chosen to delegate to someone else. It’s kind of like the ship has sailed once you’ve instantiated 100 billion of these employees to whom you’re delegating all this work. Maybe employee is kind of a weird or politically loaded metaphor. But the point is just you’ve made some collective system much more powerful than humans. One problem is you don’t have any other options. The other is that system could clearly stop you. Over time, eventually, you’re not going to be able to roll back those changes.


Prompt: Okay.

Response: Because almost all of the people doing anything in the world don’t want you to. “People” in quotes, don’t want you to roll back those changes.


Prompt: So some people think, probably what’s going to happen is one day all humans will wake up dead. You might think that it looks we’re just stuck on earth and AI systems get the whole rest of the universe or keep expanding until they meet aliens or something. What concretely do you think it looks like after that?

Response: I think it depends both on technical facts about AI and on some facts about how we respond. So some important context on this world: I think by default, if we weren’t being really careful, one of the things that would happen is AI systems would be running most militaries that mattered. So when we talk about all of the employees are bad, we don’t just mean people who are working in retail or working as scientists, we also mean the people who are taking orders when someone is like, “We’d like to blow up that city,” or whatever.


Prompt: Yep.

Response: Imagine some really rich person who on paper has a ton of money. And is asking things to happen, but they give instructions to their subordinates and then somehow nothing really ends up ever happening. They don’t know who they’re supposed to talk to and they are never able to figure out what’s happening on the ground or who to hold accountable. That’s kind of my default picture. I think the reason that I have that default picture is just because I don’t expect humans to, in cases where we fail, there’s some way in which we’re not going to really be pushing back that hard. I think if we were really unhappy with that situation then instead, you could not gradually drift off the rails, but if you really are messing up alignment then instead of gradually drifting off the rails it looks more like an outbreak of violent conflict or something like that.


Prompt: So, I think that’s a good sense of what you see as the risks of having really smart AIs that are not aligned. Do you think that that is the main kind of AI-generated existential risk to worry about, or do you think that there are others that you’re not focusing on but they might exist?

Response: So I guess the first point was, I think there are a lot of risks in the future. In some sense from our perspective what it’s going to feel like is the world accelerates and starts getting really crazy. And somehow AI is tied up with that. But I think if you were to be looking on the outside you might then see all future risks as risks that felt like about AI. But in some sense, they’re kind of not our risks to deal with in some sense, they’re the risks of the civilization that we become, which is a civilization largely run by AI systems.


Prompt: Okay. So you imagine, look, we might just have really dangerous problems later. Maybe there’s aliens or maybe we have to coordinate well and AIs would somehow be involved.

Response: Yeah. So if you imagine a future nuclear war or something like that, or if you imagine all the future progressing really quickly. Then from your perspective on the outside what it looks like is now huge amounts of change are occurring over the course of every year, and so one of those changes is that somewhere that would’ve taken hundreds of years now only takes a couple years to get to the crazy destructive nuclear war. And from your perspective, it’s kind of like, “Man, our crazy AI started a nuclear war.” From the AI’s perspective it’s like we had many generations of change and this was one of the many coordination problems we faced, and we ended up with a nuclear war. It’s kind of like, do you attribute nuclear wars as a failure of the industrial revolution, or risk of the industrial revolution? I think that would be a reasonable way to do the accounting. If you do the accounting that way there are a lot of risks that are AI risks. Just in the sense that there are a lot of risks that are industrial revolution risks. That’s one category of answer, I think there’s a lot of risks that kind of feel like AI risks in that they’ll be consequences of crazy AI driven conflict or things like that, just because I view a lot of the future as crazy fast stuff driven by AI systems.


Prompt: Okay.

Response: I’m happy to talk about some of those but I think the broad thing is just, if a lot of thinking is being done not by humans, that’s just a weird situation for humans to be in, and it’s a little bit unclear. If you’re not really thoughtful about that, it’s unclear if you’re happy with it. If you told me that the world with AI and the world without AI converged to different views about what is good, I’m kind of like, “Oh, I don’t know which of those… “ Once you tell me there’s a big difference between those, I’m kind of scared. I don’t know which side is right or wrong, they’re both kind of scary. But I am definitely scared.


Prompt: So, I think you said that relatively soon, we might end up in this kind of world where most of the thinking is being done by AI. So there’s this claim that AI is going to get really good, and not only is it getting really good, it’s going to be the dominant way we do most cognitive work, or most thinking maybe. And not only is that eventually going to happen, it’s not going to be too long from now. I guess the first thing I’d like to hear is, by not too long from now do you mean the next 1000 years, the next 100 years, the next 10 years? And if somebody’s skeptical of that claim, could you tell us why you believe that?

Response: And if you talk to someone like Eliezer Yudkowsky from MIRI, I think he would be more like “good chance that that’s only one month” or something like that between those milestones. I have the view that the best guess would be somewhere from one to five years. And I think even at that timeline, that’s pretty crazy and pretty short. Yeah. So my answer was some broad distribution over how many decades until you have AI systems that have really changed the game, and are making humans several times more productive. Say the economy’s growing several times faster than it is today. And then from there most likely on the order of years rather than decades until humans are basically completely obsolete, and AI systems have improved significantly past that first milestone.


Prompt: And can you give us a sense of why somebody might believe that?

Response: It doesn’t actually double the amount of stuff because there’s a lot of bottlenecks, but it looks like, starting from the point where AI systems are actually doubling the rate of growth or something like that, it doesn’t really seem there are enough bottlenecks to prevent further doublings in the quality of hardware or software from having really massive impacts really quickly. So that’s how I end up with thinking that the time scale is measured more like years than decades. Just like, once you have AI systems which are sort of comparable with humans or are in aggregate achieving as much as humans, it doesn’t take that long before you have AI systems whose output is twice or four times that of humans.


Prompt: Okay. And so this is basically something like, in economics you call it an endogenous growth story, or a society-wide recursive self-improvement story. Where if you double the human population, and if they’re AI systems, maybe that makes it better, there are just more ideas, more innovation and a lot of it gets funneled back into improving the AI systems that are a large portion of the cognitive labor. Is that roughly right?

Response: Yeah. I think that’s basically right. I think there are kind of two parts to the story. One is what you mentioned of all the outputs get plowed back into making the system ever better. And I think that, in the limit, produces this dynamic of successive doublings of the world where each is significantly faster than the one before.


Prompt: Yep.

Response: If output was the sum of two exponentials, one growing fast and one growing slow, then you can have a fairly quick transition as one of those terms becomes the dominant one in the expression. And that dynamic changes if humans and AIs are complementary in important ways. And also the rate of progress changes if you change… like, progress is driven by R&D investments, it’s not an exogenous fact about the world that once every year things double. But it looks the basic shape of that curve is pretty robust to those kinds of questions, so that you do get some kind of fairly rapid transition.


Prompt: Okay. So we currently have something like a curve where humanity gets richer, we’re able to produce more food. And in part, maybe not as much in wealthy countries, but in part that means there are more people around and more people having ideas. So, you might think that the normal economy has this type of feedback loop, but it doesn’t appear that at some point there’s going to be these crazy doubling times of 5 to 10 years and humanity is just going to go off the rails. So what’s the key difference between humans and AI systems that makes the difference?

Response: It is probably worth clarifying that on these kinds of questions I am more hobbyist than expert. But I’m very happy to speculate about them, because I love speculating about things.


Prompt: Sure.

Response: So I think my basic take would be that over the broad sweep of history, you have seen fairly dramatic acceleration in the rate of humans figuring new things out, building new stuff. And there’s some dispute about that acceleration in terms of how continuous versus how jumpy it is. But I think it’s fairly clear that there was a time when aggregate human output was doubling more like every 10,000 or 100,000 years.


Prompt: Yep.

Response: And that has dropped somewhere between continuously and in three big jumps or something, down to doubling every 20 years. And we don’t have very great data on what that transition looks like, but I would say that it is at least extremely consistent with exactly the kind of pattern that we’re talking about in the AI case.


Prompt: Okay.

Response: It’s good not to be dying because you’re hungry. But that means that you no longer have this loop of more output leading to more people. I think there’s a second related explanation, which is that the world now changes kind of roughly on the time scale of human lifetime, that is like, it now takes decades for a human to adapt to change and also decades for the world to change a bunch. So you might think that changing significantly faster than that does eventually become really hard for processes driven by humans. So you have additional bottlenecks just beyond how much work is getting done, where it’s at some point very hard for humans to train and grow new humans, or train and raise new humans.


Prompt: Okay.

Response: So those are some reasons that a historical pattern of acceleration may have recently stopped. Either because it’s reached the characteristic timescales of humans, or because we’re no longer sort of feeding output back into raising population. Now we’re sort of just growing our population at the rate which is most natural for humans to grow. Yeah, I think that’s my basic take. And then in some sense AI would represent a return to something that at least plausibly was a historical norm, where further growth is faster, because research is one of those things or learning is one of those things that has accelerated. Recently I don’t know if you’ve discussed this before, but Holden Karnofsky at Cold Takes has been writing a bunch of blog posts summarizing what this view looks like, and some of the evidence for it. And then prior to that, Open Philanthropy was writing a number of reports looking at pieces of the story and thinking through it, which I think overall taken together makes the view seem pretty plausible, still.


Prompt: Okay. So I’m now going to transition a little bit to questions that somebody who is very bullish on AI x-risk might ask, or ways they might disagree with you. I mean bullish on the risk, bearish on the survival. Bullish meaning you think something’s going to go up and bearish meaning you think something’s going to go down. So yeah, some people have this view that it might be the case that you have one AI system that you’re training for a while. Maybe you’re a big company, you’re training it for a while, and it goes from not having a noticeable impact on the world to effectively running the world in less than a month. This is often called the Foom view. Where your AI blows up really fast in intelligence, and now it’s king of the world. I get the sense that you don’t think this is likely, is that right?

Response: I think that’s right. Although, it is surprisingly hard to pin down exactly what the disagreement is about, often. And the thing that I have in mind may feel a lot like foom. But yeah, I think it’s right, that the version of that, that people who are most scared have in mind, feels pretty implausible to me.


Prompt: Okay. So, it seems like key place where that story differs is in the foom story, it was very localized. There was one group where AI was growing really impressively. Am I right, that you are thinking, no, probably a bunch of people will have AI technology that’s like only moderately worse than this amazing thing?

Response: Yeah. I think that’s basically right. The main caveat is what “one group” means. And so I think I’m open to saying, “Well, there’s a question of how much integration there is in the industry.”


Prompt: Yeah.

Response: And you could imagine that actually most of the AI training is done… I think there are these large economies of scale in training machine learning systems. Because you have to pay for these very large training runs, and you just want to train. You want to train the biggest system you can and then deploy that system a lot of times, often. Training a model that’s twice as big and deploying half as many of them is better than training a smaller model and deploying. Though obviously, it depends on the domain. But anyway, you often have these economies of scale.


Prompt: Yep.

Response: If you have economies of scale, you might have a small number of really large firms. But I am imagining then you’re not talking, some person in the basement, you’re talking, you have this crazy $500 billion project at Google.


Prompt: Yep.

Response: In which Google, amongst other industries, is being basically completely automated.