# 1-page outline of Carlsmith's otherness and control series

*Joe’s summary is *[*here*](https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi)*, these are my condensed takeaways in my own words. All links in this section are to the* [*essays*](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W)*. *

Outline
-------

*   Carlsmith tackles two linked questions:
    *   How should we behave towards future beings (future humans, AIs etc)?
    *   What should our priors be about how AIs will behave towards us?
*   So the first question:
*   We worry about value in the future, how should we behave towards the future and future beings?
    *   (1) [We could trust in some base goodness - the universe, god, AIs being good](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/sJPbmm8Gd34vGYrKd)
    *   (2) We could accept that all future beings will be alien to us and stop worrying ([here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/aBmBaL95hrNLvqJse) and [here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/cnv5g7jCWLw9LYKxa))
    *   (3) [We could have moral systems or concepts of goodness/niceness](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/rdTgtHn3neGzkyCrL)
    *   (4) We could seize power over the future ([here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/fPajcaopbQ7pxztf4) and [here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/oJRMDAvWk8diDjNsG))
    *   (5) We could adopt a different poise which centred around notions like growth/harmony/ “attunement” ([here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/gvNnE6Th594kfdB3z) and [here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/us8cqwudP5sePqWM2))
*   (1) 
    *   [Yudkowskians don’t tend to trust in god or the universe](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/sJPbmm8Gd34vGYrKd#The_death_of_many_gods). That’s part of their schtick. [They don’t trust AIs will be good](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/sJPbmm8Gd34vGYrKd#Are_moral_realists_theists_)
*   (2) 
    *   [This is perhaps what Hanson argue for](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/aBmBaL95hrNLvqJse#Will_future_humans_be_more_different_from_us_than_AI_risk_expects_)[^rzxcj172aum]
    *   Yudkowskians do not buy this
        *   [Are they avoiding how alien future people will be?](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/aBmBaL95hrNLvqJse#Will_future_humans_be_more_different_from_us_than_AI_risk_expects_) 
        *   How can they justify a notion of good that’s robust over time? ([eg often Yudkowsky pushes against this notion](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/cnv5g7jCWLw9LYKxa#The_fragility_of_value)) 
        *   [Are they avoiding discomfort?](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/aBmBaL95hrNLvqJse)[^bznvmdrtdvb]
*   (3) 
    *   [Moral systems vary wildly at edge cases](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/cnv5g7jCWLw9LYKxa#The_fragility_of_value)
    *   On the scale of the whole future, even [good people](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/rdTgtHn3neGzkyCrL#Boundaries) controlling it might be a moral disaster. The notion of paperclipping elides this, because it is involuntary and aesthetically dull. [But the arguments also extend to law abiding and even relatively joyful beings](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/rdTgtHn3neGzkyCrL#Is_niceness_enough_)
*   (4) 
    *   [This powerseeking is hard to distinguish from the power seeking of AIs](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/rdTgtHn3neGzkyCrL#Utilitarian_vices:~:text=Utilitarianism%20does%20not%20love%20you%2C%20nor%20does%20it%20hate%20you%2C%20but%20you%27re%20made%20of%20atoms%20that%20it%20can%20use%20for%20something%20else.)[^abcksbm7jr] (also ([here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/fPajcaopbQ7pxztf4) and [here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/oJRMDAvWk8diDjNsG))
*   (5)
    *   [There is some notion of attunement/ trust/ growth/ balance](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/gvNnE6Th594kfdB3z#Gestures_at_green) which [rationalists and EAs are quite inimical to](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/gvNnE6Th594kfdB3z#Gestures_at_green:~:text=the%20non%2Dgreen%20of%20both%20stands%20out)[^45gcyo4qvmm]
    *   [There seems some way of being](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/us8cqwudP5sePqWM2#Being_your_soul) which navigates how to interact with complex systems, letting them grow, tending to them without being able to dominate the or being dominated by them
    *   [What might this look like?](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/us8cqwudP5sePqWM2)
*   Second question. [How might AIs treat us](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL)?
    *   If we assume that we cannot trust (not 1,3) think it’s a problem (not 2)  and ignore (5) then 
        *   it’s very easy to see AI as a tool while it is weaker than us and a competitor if it becomes stronger
        *   Even if AI may be others ways than this, the safe option is to assume the competitor frame
    *   Under (5) AI might be 
        *   [Something good but alien (like an octopus)](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#Gentleness)
        *   Something dangerous but not competitive ([like a dead-eyed bear](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#Getting_eaten))
        *   Something else. Other. Not us
*   Takeaways
    *   [We should consider other ways to be towards those we could control but who might control us](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/us8cqwudP5sePqWM2#Being_your_soul)
    *   [We should consider other relationships towards AI](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#Getting_eaten:~:text=Fear%3F%20Oh%20yes%2C%20I%20expect%20fear.%20But%20not%20only%20that.)[^jefd70qm9q]
    *   In AI discourse there is a lack of clarity in notions of attunement, respect, harmony in relation to the sub-optimal choices of other conscious beings ([here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/gvNnE6Th594kfdB3z#Green__according_to_non_Green) and [here](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/us8cqwudP5sePqWM2))
    *   [It is possible that our priors are driven by our lack of this notion](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#When_species_meet)[^vxt7x3i14xf][^xup0f2dli9c]

Some quotes I liked/was moved by:
---------------------------------

*Where Joe is quoting someone else I also link to the original source*  
 

On being [‘just statistics’](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#When_species_meet:~:text=%E2%80%9CJust%E2%80%9D%20is%20rarely,depression%2C%20sociopathy%2C%20etc.)

> “Just” is rarely a bare metaphysic. More often, it’s also an aesthetic. And in particular: the aesthetic of disinterest, boredom, deadness. Certain frames – for example, mechanistic ones – prompt this aesthetic more readily. But you can spread deadness over anything you want, consciousness included. Cf depression, sociopathy, etc.

  
Werner Herzog, on the deadness of nature: ([source](https://www.youtube.com/watch?v=iWYaC5YBaJk), link to [essay section](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/mzvu8QTRXdvDReCAL#People_in_bear_costumes:~:text=that%20the%20bears%2C%20at%20least%2C%20do%20not%20bow%20back%3A))

> “And what haunts me is that in all the faces of all the bears that Treadwell ever filmed, I discover no kinship, no understanding, no mercy. I see only the overwhelming indifference of nature. To me, there is no secret world of the bears, and this blank stare speaks only of a half-bored interest in food.”

From Yudkowsky: ([source](https://secularsolstice.github.io/speeches/gen/Nihil_Supernum.html), link to [essay section](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/sJPbmm8Gd34vGYrKd#:~:text=Still%2C%20though%3A%20what%20if%2C%20actually%2C%20Ultimately%2C%20we%20are%20orphans%3F%20Yudkowsky%20has%20a))

> No rescuer hath the rescuer.
> 
> No Lord hath the champion,
> 
> no mother and no father,
> 
> only nothingness above.

Yudkowsky, on the death of his brother[^bsuji7r964u], ([source](https://www.yudkowsky.net/other/yehuda), link to [essay section](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/sJPbmm8Gd34vGYrKd#:~:text=written%20when%20his%20brother%20Yehuda%20died%3A))

> ... Yehuda did not "pass on". Yehuda is not "resting in peace". Yehuda is not coming back. Yehuda doesn't exist any more. Yehuda was absolutely annihilated at the age of nineteen. Yes, that makes me angry. I can't put into words how angry. It would be rage to rend the gates of Heaven and burn down God on Its throne, if any God existed. But there is no God, so my anger burns to tear apart the way-things-are, remake the pattern of a world that permits this....

> Haters gonna hate; atheists gonna yang[^7vi35w1ow1h]; agents gonna power-seek

([link](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/fPajcaopbQ7pxztf4#:~:text=Haters%20gonna%20hate%3B%20atheists%20gonna%20yang%3B%20agents%20gonna%20power%2Dseek))  
 

> Utilitarianism does not love you, nor does it hate you, but you're made of atoms that it can use for something else 

([link](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/rdTgtHn3neGzkyCrL#:~:text=Utilitarianism%20does%20not%20love%20you%2C%20nor%20does%20it%20hate%20you%2C%20but%20you%27re%20made%20of%20atoms%20that%20it%20can%20use%20for%20something%20else))

On whether it is [wrong to cut down ancient trees?](https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/gvNnE6Th594kfdB3z#:~:text=And%20yet%2C%20for%20all%20this%2C%20something%20about%20just%20cutting%20down%20this%20ancient%2C%20living%20tree%20for%20lumber%20does%2C%20indeed%2C%20feel%20pretty%20off%20to%20me.%20It%20feels%2C%20indeed%2C%20like%20some%20dimension%20related%20to%20%22respect%22%20is%20in%20deficit.)

> And yet, for all this, something about just cutting down this ancient, living tree for lumber does, indeed, feel pretty off to me. It feels, indeed, like some dimension related to "respect" is in deficit.

* * *

Crossposted from [my blog](https://nathanpmyoung.substack.com/) (link to be attached later because I formatted for lesswrong before substack) 

  
What did you think of this? I have a much longer point by point summary and if 10 people sign up to a [paid subscription](https://nathanpmyoung.substack.com/subscribe) of my blog I’ll finish and post that to them[^refwgycbkbn].

Are there other pieces you would like summarised/translated into Nathanese?

[^rzxcj172aum]:  It’s funny to me that Carlsmith’s hierarchy of atheism seems to imply Hanson is the deepest atheist, disbelieving not only in God and the goodness of the universe but also that there is a stable notion of good over time. I softly endorse this 

[^bznvmdrtdvb]: Specific quote: "On the other hand, some sort of discomfort in trying to control the values of future humans persists (at least for me). I think Hanson is right to notice it – and to notice, too, its connection to trying to control the values of the AIs. I think the AI alignment discourse should, in fact, prompt this discomfort – and that we should be serious about understanding, and avoiding, the sort of yang-gone-wrong that it's trying to track." 

[^abcksbm7jr]: Specific quote: "Utilitarianism does not love you, nor does it hate you, but you're made of atoms that it can use for something else." 

[^45gcyo4qvmm]: Specific quote: "Indeed, for closely related reasons, when I think about the two ideological communities that have paid the most attention to AI risk thus far—namely, Effective Altruism and Rationalism—the non-green of both stands out." 

[^jefd70qm9q]: Specific quote: "Fear? Oh yes, I expect fear. But not only that. And we should look ahead to the whole thing." 

[^vxt7x3i14xf]: Specific quote: "I want to start this series by acknowledging how many dimensions of interspecies-relationship this narrative leaves out" 

[^xup0f2dli9c]: To me, there is a slight undercurrent of this being a self-fulfilling prophecy/ vicious cycle- that we make a world of conflict slightly more likely by considering that world more likely than it is 

[^bsuji7r964u]: I find this quote tremendously moving. And some part of me sings in unison 

[^7vi35w1ow1h]: Carlsmith links the notion of powerseeking, agency, activity and a lack of trust and labels it 'yang'. I have thought using it a lot since 

[^refwgycbkbn]:  And write more of this kind of stuff in future. This post took 5 - 15 hours more than if I'd just listened to the pieces. Getting it this short took a long time, as the saying goes, "If I had more time, I would have written a shorter letter" (seems we don't know who originally said this)