# IABIED and Memetic Engineering

Probably no one here needs another review of the *content* of [the
book][iabied]. If you do, try [Scott’s][acx] or [Zvi’s][zvi]; I don’t
think I have anything to say on that front that they haven’t.

I do have a few thoughts on its presentation that I haven’t seen
elsewhere.[^meta] It made me think about target audiences, and opportune
moments, and what Eliezer and Nate’s writing process might look like.

*If Anyone Builds It, Everyone Dies* is not *for* me, except in the
sense that it goes nicely on my shelf next to Bostrom’s
*Superintelligence*. I don’t need to be convinced of its thesis, and
I’ve heard its arguments already. I’m admittedly less confident than
Eliezer, but most of my uncertainty is model uncertainty, so that’s as
it should be; right or wrong, someone who’s spent their whole life
studying this problem should have *less model uncertainty than me*.

The writing style is less...Eliezer, than I expected. It’s noticeably
simplified relative to Eliezer’s usual fare. It’s not talking to the
general public, exactly, I don’t think you *could* simplify it that far,
but decision-makers among the general public. +1.5 SD instead of +2.5.
It reminds me of a story about Stephen Hawking’s experience writing A
Brief History of Time, wherein his editor told him that every equasion
in the book would cut sales in half; this feels like someone told
Eliezer that every prerequisite-recursion would cut its reach in half.

I wonder what the collaborative process was like, who wrote what.
Eliezer’s typical writing is...let’s go with “abrasive.” He thinks he’s
smarter than you, he has the chutzpah to be right about that far more
often than not, and he’s unrepentant of same, in a manner that outrages
a large fraction of primates. That tone is entirely absent from IABIED.
I wonder if a non-trivial part of Nate’s contribution was “edit out all
the bits of Eliezer’s persona that alienate neurotypicals,” or if some
other editor took care of that. I’m pretty sure *someone* filtered him;
when, say, the Example ASI Scenario contains things like (paraphrased)
“here’s six ways it could achieve X; for purposes of this example at
least one of them works, it doesn’t matter which one” I can practically
hear Eliezer thinking “...because if we picked one, then *idiots* would
object that “method Y of achieving X wouldn’t work, therefore X is
unachievable, therefore there is no danger.” And then I imagine Nate (or
whoever) whapping Eliezer’s key-fingers or something.

Or maybe he’s just mellowed out in the years since the Sequences. Or
maybe he’s filtering himself because [offending people is a bad way to
convince them of things][losing] and that is unusually important right
now; if he was waiting for the opportune moment to publish a Real Book,
best not to waste that moment on venting one’s spleen.

Maybe all three went into it. I don’t know. The difference just jumped
out at me.

Scott criticizes the Example ASI Scenario as the weakest part of the
book; I think he’s right, it might be a reasonable scenario but it
*reads* like sci-fi in a way that could easily turn off non-nerds. That
said, I’m not sure how it could have done better. The section can’t be
omitted without lossage, because fiction speaks to the intuitive brain
in a way that explicit argument mostly can’t. It can’t avoid feeling
like sci-fi, because sci-fi got here first. And it can’t avoid feeling
like nerd-fi, because an attempt to describe a potential reality has to
justify things that non-nerd fiction doesn’t bother with.

(...I suddenly wonder if there is a hardcore historical-fiction subgenre
similar in character to hard SF; if so, the thing they have in common is
the thing I’m calling ‘nerd-fi’, and I expect it to turn off the
mainstream)

I’ve heard a couple complaints that the title “If Anyone Builds It,
Everyone Dies” is unacceptably sensationalized. The complaint made me
think about something [Scott Alexander said about books vs.
blogging][books]:

> An academic once asked me if I was writing a book. I said no, I was able
> to communicate just fine by blogging. He looked at me like I was a
> moron, and explained that writing a book isn’t about communicating
> ideas. Writing a book is an excuse to have a public relations campaign.
> [...] The book itself can be lorem ipsum text for all anybody cares. It
> is a ritual object used to power a media blitz that burns a paragraph or
> so of text into the collective consciousness.

I think burning in a whole paragraph might be too optimistic. If you
want to coordinate millions of people, [you get about five
words][words]. In this case, the five words have to convey the
following:

* That, if built, ASI would almost-certainly *kill everyone*.
* That it doesn’t matter *who* builds it, you still die.

“If Anyone Builds It, Everyone Dies” seems about as short as you can get
while still getting those points across. It doesn’t fit in five words,
but it comes close, and every one of those words does necessary work.
Its tone is sensationalist, but I see no way to rephrase it to escape
that charge without sacrificing either meaning or transmissability. It
doesn’t *quite* fully cover the points above -- the “it” is ambiguous --
but I don’t see an obvious way to fix that flaw either. The target
audience of the title-phrase mostly won’t know what “ASI” means.

(I feel like I’m groping for a concept analogous to an orthogonal basis
in linear algebra -- a concept like “the minimal set of words that span
an idea” -- and the title “If Anyone Builds It, Everyone Dies” *almost*
gets there)

All of which is to say that the title alone *might* compress the
authors’ thesis enough to fit in the public’s medium-term memory, and
I’m fairly sure that was deliberate. The book’s content was written to
convince “the sort of people the public listens to” (i.e. not nerds, but
readers), while its title was (I suspect) chosen to stick in the mind of
the actual public. The content is the message, the title is the meme.

Most books don’t have to care about any of that. Most books can just
choose an artistic title that doesn’t fully cover the underlying idea,
because that’s what the rest of the book is for. If you’re aiming for
memetic transmission beyond your readers, though, your options are more
constrained.

I notice that it would be difficult to identify the book, even to mock
it, without spreading the meme. Was that part of the idea, too? I’m not
on social media, but I presume the equivalent of the Sneer Club is
working overtime.

I have mixed feelings about this whole line of thought. I’m quite sure
Eliezer and Nate would have *moral qualms* about a media blitz backed by
literal or metaphorical lorem ipsum. So they write in good faith, as
they should. But it’s no secret that the book is a means, not an end. If
you believe your case, but you know most of the people you aim to reach
won’t bother reading the case, just the meme, does it still count as
propaganda?

One ought to *convince* people via valid arguments they can check for
consistency and correctness, not *persuade* them via meme propagation
and social contagion. How does one adhere to that code, when approaching
the general public, who think in slogans and vibes and *cannot* verify
any argument you make, even if they were inclined to do so?

How do you *not* compromise on epistemic integrity when you only have
five words to work with and can’t afford to walk away?

The book’s implicit answer is “pack as much meaning as possible into the
[explicit content][explicit] of the meme, and bundle the meme with the
argument so at least the good-faith version is *available*.” I suppose I
can’t argue with that answer. I certainly don’t have a better one.

[acx]: https://www.astralcodexten.com/p/book-review-if-anyone-builds-it-everyone
[zvi]: https://thezvi.substack.com/p/book-review-if-anyone-builds-it-everyone
[books]: https://www.astralcodexten.com/p/book-review-what-we-owe-the-future
[words]: https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-get-about-five-words
[iabied]: https://www.amazon.com/dp/0316595640
[validity]: https://www.lesswrong.com/posts/WQFioaudEH8R7fyhm/local-validity-as-a-key-to-sanity-and-civilization
[explicit]: https://benjaminrosshoffman.com/explicit-content/
[losing]: https://www.lesswrong.com/posts/6yTShbTdtATxKonY5/how-to-not-lose-an-argument

[^meta]: *Meta: I’m not on social media, so I could easily have missed
things elsewhere.