# An Attempt to Explain my AI Risk Explainer Attempt

*Portions of the blog post relevant to LW reproduced below. Portions of the post explaining AI risk to general audiences or pointing to other AI risk resources omitted. My LW-specific summary would be: aggressive text folding is really appealing to some people, but appears generally unpopular.*

I spent much of 2023-2025 writing an explainer about AI risk, with what I hoped would be a unique twist. However, I discovered foundational problems with the unique twist that led me to abandon the work.

*... introduction to AI risk omitted ...*

There are already many AI risk explainers: what was the unique thing this project was trying to do?
===================================================================================================

There are two related problems with explaining AI risk through written formats, which this post calls the detail dilemma:

*   People do not want to read long winded explainers full of detailed arguments. Too long, didn't read.
*   Readers of short explainers can easily come up with counter-arguments ("an AI is doing bad things? Why can't we just turn the computer off?"), but because the work is short, it does not have room to address every detail and possible counter-argument. If a reader starts out skeptical, the failure to address their specific arguments may be enough to cause them to reject the entire concept of AI risk as unfounded.
*   Adding enough detail to address everyone's potential arguments brings us back to the first problem with long explainers...

My approach to threading the detail dilemma was to craft a new writing format with expandable levels of detail. As an brief completely serious example:

+++ Cats should rule the earth.

+++ Humans have done a poor job of running the Earth; cats could hardly do a worse job.

(Here I would tally up the sins of the humans; on the other hand, we could only list such cat sins such as "broke the vase", "barfed on the sofa", and "caused me to trip while carrying something heavy".)




++++++ Cats already believe they rule the earth, so they will be natural rulers.

(I would include anecdotes here, like demanding to be let into a room and then immediately demanding to be let out of the same room.)

(Any reports that cats do not possess sufficient authority to be fed on demand are slander.)




++++++ Cats are too cute, and therefore deserve to set all aspects of government policy.

(I would include photos here, to remind people of cat cuteness.)

(The link between cuteness and policy making ability should be obvious.)




+++


+++

A longer example with additional effort would be [my argument about why AI War (conflict between AIs) would not prevent AI risk](https://thenoviceoof.com/ai/war/ai_war_v1_0.html)[^apv1u403uof].

I figured a format built around nested foldouts (henceforth referred to as the foldout format) would have some nice properties:

1\. A Reader Driven Deep Dive
-----------------------------

Readers could choose where they wanted to dive into the details (a sort of [choose your own adventure](https://en.wikipedia.org/wiki/Choose_Your_Own_Adventure)); they could skip arguments if we agreed, and dive deep where we disagreed.

My understanding is that skilled readers already do this with long texts (like a PhD thesis): why read the entire thing if only one part is relevant to your interests? Instead of passively following the text, one reads more actively, using existing sign posts like headings and paragraph breaks to skip to the most interesting material.

However, not everyone is a skilled reader. Perhaps it is good if the writing format itself forces the reader into an active reading stance. I could argue that the foldout writing system does this, since the reader has to actively decide whether to follow any particular line of thought into a foldout.

2\. A Structured Deep Dive
--------------------------

Writing is usually linear, but we readers need to recover the structure of the argument the writer had in mind from this linear stream: "Ok, this sentence sure seems like the local thesis. This paragraph is addressing an argument about the thesis. This page is touching on some minor detail related to the thesis. This paragraph seems to be tying the thesis to a previou- oh, never mind, it's actually making a new point." Good writing makes this reconstruction by the reader natural and easy.

The foldout format makes this structure explicit, hiding subordinate arguments and details by default, and making it clear which content is related. As an example, a reader may be deep into my discussion ["are centrally hosted AIs at a disadvantage relative to locally hosted AIs in a war setting?"](https://thenoviceoof.com/ai/war/ai_war_v1_0.html#details-0094) This deep into the weeds the reader may realize they've lost the thread about how this ties into general AI risk. In a linear text, they would need to find the last heading, or some heading before that; heavens help you if the text is a book with sparse headings and obfuscatory chapter titles[^6ukparws6it]. With the foldout format, it is easy to collapse parent foldouts until one sees a larger point like "multipolar AI is probably not stable".

Structure also seems like an advantage over something like a [RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) chatbot; a linear conversation can dive deep and answer specific questions, but it is not easy to keep track of the [stack](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)) of topics in a conversation. Because of this, it seems like it is easy for a conversation to get lost in the details.

So in theory an explicit foldout structure makes it easier to comprehend a complicated argument.

3\. A Comprehensive Deep Dive
-----------------------------

Normal writing is subject to the detail dilemma: writers need to balance including details against losing readers with increasingly long-winded writing. Student writers need to be taught to ["kill your darlings"](https://slate.com/culture/2013/10/kill-your-darlings-writing-advice-what-writer-really-said-to-murder-your-babies.html); their clever turns of phrase, their devastating sub-sub-sub-arguments, their beautiful details need to be measured against whether they're serving the reader. If the target audience is bored or confused, the detail has to go. Even if a single studious reader is served by a clarification, is is probably a mistake to include if \*only\* that reader is served by that aside.

The foldout format jettisons the need for each detail to serve all audiences, since each reader will decide for themselves how deep they want to go, allowing adding many more details than otherwise possible[^u000cx5fpvf]. If properly written, one document could serve an audience looking for something short and punchy, but also serve audiences looking for a comprehensive deep dive with all applicable details[^mk4nahip04l].

Why abandon the project?
========================

There were a number of problems that cropped up:

People didn't like the format
-----------------------------

Many early draft readers explicitly wanted a more traditional long form writing format, which boiled down to removing the foldouts[^mdd81f2daad]. This isn't fatal to the project, perhaps it just needed more work, but I heard only minimal feedback from people that liked the format[^ws9zatj79kb]. Without such a positive signal, it is difficult to plow past the negative feedback.

My hypothesis is that the format is widely disliked, but there are a few people who really love this sort of thing, including me. I have a few ideas about why people generally don't like the format:

### Theory 1: Active reading is hard

I theorized earlier that the format tends to force readers into an active reading stance. However, "forcing" your readers to do anything is probably bad. Great writing entices readers to engage[^6xcjqegt06q]. Being forced into active reading sounds much more adversarial, not something you want in the relationship between writer and reader.

An alternative frame is that ["choices are bad"](https://thezvi.wordpress.com/2017/07/22/choices-are-bad/), and the foldout format floods the reader with choices around whether to open a foldout or not. Therefore, the format is putting stress on the reader, and should be considered bad.

### Theory 2: Outlines are unpopular

The foldout format is naturally suited to outlines: each foldout is summarized by a short claim, backed up at length within the foldout, all structured within a hierarchy of foldouts. However, most of our communication is not through outlines, but through prose. We could argue about why, but it seems clear to me that people don't like outlines, and I strongly suspect at least some of that dislike spills over to the foldout format, with at least one piece of feedback explicitly asking for more prose-like writing. It might be possible to write for the foldout format in a style closer to prose, but I haven't put in the work to figure out how.

* * *

In retrospect, there already exists something similar to the foldout format, [argument mapping](https://en.wikipedia.org/wiki/Argument_map). The argument mapping platform Kialo has a similar tree structure and mechanism to unfold high level arguments into details (as an example, see [this argument about AI risk](https://www.kialo.com/mitigating-the-risk-of-extinction-from-ai-should-be-a-global-priority-alongside--risks-such-as-pandemics-and-nuclear-war-63178)). However, argument mapping is a niche interest, even with a long history and luminary attention, [including Engelbart in 1962](https://en.wikipedia.org/wiki/Argument_map#Development_of_computer-supported_argument_visualization), father of ["the mother of all demos"](https://en.wikipedia.org/wiki/The_Mother_of_All_Demos). Engelbart's [computer mouse](https://en.wikipedia.org/wiki/Computer_mouse) concept clearly succeeded, while for whatever reason argument mapping clearly didn't. I think this comports cleanly with my notion that this idea really appeals to a small number of weirdos like me, but is otherwise broadly unpopular.

There were other problems with the format.

Kitchen Sink Syndrome
---------------------

As mentioned previously, the foldout format makes it easy to include all possible detail. This is great for making a work comprehensive, but then I needed to write all those details up, which takes time. In the drive to be comprehensive, I included everything, which meant it took a lot of time to write things up, including details that might not matter to anyone. This was exacerbated because…

Writing Is Not My Thing
-----------------------

One major problem was that I don't actually like writing[^m2xuonqdtn9]. You would think I already knew this, since I have taken 2 multi-year hiatuses from blogging, but I figured that I could power through it. I thought the topic was important, and maybe I was creating something new in the fundamental field of writing! To be fair, these reasons were enough to motivate me to regularly write for the last 2 years, and quitting my job[^pnekf5z77f] gave me the time, but as it turns out 2 years was not enough time to finish the writing I wanted to do.

Phone Mode
----------

Another piece of feedback I received multiple times regarded how the format handles phones. The central problem is that the format concerns itself with deeply hierarchical information. A natural way to display hierarchy is to use indentation, but the indents need to be large to be legible. Modern phone screens are narrow, which means deeply nested legible indents will quickly eat up all available horizontal space, in some cases pushing details off the screen[^6zdr2mrwd2u]. The solution I eventually settled on was to simply have each foldout take over the screen to maximize real estate, which also had a sweet and pleasingly simple implementation ([Github](https://github.com/thenoviceoof/outline-folder)).

Unfortunately, phone beta readers hated the full screen take over. Before I ended the project, I was seriously considering telling phone readers to use a larger screen, perhaps even blocking them instead of letting them wade through a sub-optimal reading experience. On the other hand, winning in the marketplace of ideas means meeting people where they are, and the people are on their phones.

What a cruel world. A major perk to abandoning the project is not having to resolve this problem.

Lack of expected feedback
-------------------------

I was surprisingly disheartened by the lack of feedback from the wider AI risk community. I released a [small incremental portion](https://www.lesswrong.com/posts/Ymc3tQ2LL8cdCMMRB/linkpost-ai-war-seems-unlikely-to-prevent-ai-doom) of the larger project to LessWrong (LW), which I forecast as not interesting for an LW audience, so it would be fine if it got little traction and attention. However, putting up the post and actually getting little traction and attention as expected[^9yg26vzh8ut] was in fact much more disheartening than expected.

Perhaps one component of the disappointment was that there appeared to be little interest in improving AI risk communications from *the* community that would be most interested in it, and if there is no interest there, then where? One might say that it's just because I wrote something beyond salvaging, but the AISafety.info folks in the same time frame were also [trying to get feedback and also received minimal feedback](https://www.lesswrong.com/posts/ZLuwGys2du8Difumu/community-feedback-request-ai-safety-intro-for-general)[^k0dbkuzxv8]. Perhaps there's something about reading with intent to give feedback that repels even sophisticated readers[^fl9pfizlrz]: I myself didn't give feedback to the AISafety.info folks[^bkw8tfgi4b]. How could I blame other readers for doing the same to me?

All this said, being disheartened (even unexpectedly) wasn't necessarily a bad thing: it did get me to re-evaluate my assumptions and decide the project wasn't worth more time and effort.

Other Minor Problems
--------------------

+++ Best practices for writing need adaptation to foldouts.

There's an existing body of best practices for writing straight prose, which seems difficult to transfer directly to the foldout format. For example, since a foldout can be open or closed, foldout text needs to accommodate multiple different continuations. This works in a pure disjointed outline where nothing is expected to flow, but seems challenging with prose.




++++++ Talking about works using extensive foldouts introduces ambiguity.

Usually saying "I read X" is pretty clear: someone has looked at and comprehended all the words contained in X, even if they have a different interpretation or have forgotten some details. However, saying "I read Y, which uses a foldout format" introduces ambiguity: did you read \*everything\*? The format implicitly encourages you to skip parts of it, so there's much less shared context when talking about Y. This also showed up in practice when discussing with beta readers, when I would need to clarify which parts of the work the reader had actually read. This seems like a minor annoyance, but being annoying to talk about is probably not good for the popularity of a work.




++++++ Common topics don't necessarily fit into trees of foldouts.

The format assumes that arguments and details can be arranged in a tree, but this is not always possible. For example, arguments about AI consciousness should be included in discussions both about AI intelligence and AI volition, because some people believe that consciousness is important for intelligence or volition. I worked around this by having a discussion of AI consciousness in one location and then linking to it in other locations it came up, but other more densely interconnected content might work better with a wiki format.




+++

Things I should have done differently
=====================================

In retrospect, I should have done more nothing.

After I quit my job, I was burnt out. I think this goes some way to explaining why I wasn't strategic. I only took a break for a month after quitting, and then started grinding away at writing. Not much time for mental recovery after a decade at the grindstone[^kzd1qzfq7u].

If you're really interested in my thoughts about how I could have been more strategic, I have shoved those into foldouts:

+++ Project planning may have helped me keep the bigger picture in mind.

Project planning may have helped discover future road blocks, or highlight that I should come up with ways to test the format. For example, in retrospect I could have tried to release the 1st section of the document stand alone, and probably gotten the same feedback as above, but sooner. It wasn't obvious I could do this at the start, but if I had built in regular checkpoints (another project planning tool!), I could have noticed sooner than later and tried it out.




++++++ I could have tried to use a more social intensive approach to writing, keeping things focused on usefulness for real readers.

A more socially intensive approach to building the document would have been to create a minimal outline of the major arguments, and then start fleshing it out, but only with arguments and chains of reasoning found "in the wild" from discussions with people from all sides. This would have prevented some of the potential navel gazing problems I mentioned earlier, and may have highlighted problems with the foldout format earlier from intensive feedback. However, I'm not sure I could have pulled this off even without any burn out problems: telling the introvert to talk to everyone all the time sounds like a recipe for disaster. It's even worse than this, perhaps something like "tell the introvert to get into Twitter arguments with everyone all the time".




++++++ I could have tried to separate proving out the foldout format from writing about AI risk.

One other thing I could have tried was to separate out testing the foldout format from the AI risk writing. For example, I could have adapted an existing set of arguments into the foldout format, like [this climate FAQ](https://skepticalscience.com/argument.php?f=taxonomy), and gotten feedback about whether the format was useful or not. The problem is that climate change is highly politicized, which may have made feedback hard evaluate, and I don't know of any other similar compilations of arguments that would be easy to convert. On the other hand, doing this may have gotten me to wonder why existing argument mapping software was not already popular.

I think AI risk is becoming politicized, but even today it doesn't seem as politicized as climate change. Maybe give it a few years?




+++

In summary, the foldout format and ideas like it are compelling traps, which probably don't work for general communication[^9qwk8fyw8ju].

*... Lists of existing AI explainers omitted ...*

[^apv1u403uof]: I used AI Doom to refer to AI Risk in that post, but decided to use AI Risk here. Maybe I really am not cut out to be a writer? 

[^6ukparws6it]: For example, simple numbered chapters like "Chapter 5", or broad chapter title like "On Warfare", or obfuscatory titles like "The Hungry Mongoose Leaps into the Urn" which is nonsensical until you read 80% of the chapter and it turns out to be a reference to a clever quote that perfectly sums up the chapter. People never forget things, so the title is perfect. 

[^u000cx5fpvf]: That said, adding additional detail is not completely cost less: adding more details to a deeply nested foldout will add more cognitive load to readers who reach that foldout. However, in pop comp-sci terms it'd be more like O(ln N) instead of O(N) in cognitive load for picky readers. 

[^mk4nahip04l]: I was especially interested in writing a resource that would cover a variety of basic assumptions. For example, a major point of contention around AI risk is whether intelligence is actually important for humans. Instead of only assuming one viewpoint or the other, I wanted to present both cases, and argue that they both lead to AI risk. This sort of subjunctive juggling would normally be a huge mess, but being able to shunt the mess into a foldout would allow the high level argument to stay coherent and cohesive. 

[^mdd81f2daad]: I'm reading between the lines of feedback similar to "can I simply view everything?" That is, the foldouts were annoying enough to request some way to work around them, but not so annoying that readers could pinpoint "X is super annoying" or "X is annoying for Y reason". 

[^ws9zatj79kb]: While working on the explainer I didn't hear any positive feedback. After killing the project but before I finished writing up this post, I did hear from *a* beta reader that they liked the format. It's possible that there's more silent foldout format enjoyers, but it still seems compatible with my theory that the format only appeals to a small number of people. 

[^6xcjqegt06q]: One way this isn't true is if you've heard about a difficult work by word of mouth. In this case the reader might have a terrible experience, but they are willing to put in the work on the word of an outside authority. It is technically possible this could have happened to my work, but it's nothing to bet on. 

[^m2xuonqdtn9]: maybe I should have spent the last 2 years trying to induce a "virtuous cycle" similar to Scott Alexander. 

[^pnekf5z77f]: I didn't quit my job just to write, but it was an upside: I finally had time to do that writing I wanted to do! Or, you could be a bit more cynical and say I needed more time to hype myself up and overcome my natural dislike of writing. 

[^6zdr2mrwd2u]: It is possible to ask phone readers to turn their phones to landscape, except #1 that's kind of a big ask #2 now you can barely see anything vertically. 

[^9yg26vzh8ut]: Long time LW readers might object that the post didn’t do all that badly. At time of writing (2025-09-17) the post was at +7 karma and had 6 comments, which is a pretty good amount of engagement, more than many other posts get. As an example, I would estimate that half of the recent posts on LW have no comments at all. However, I was mostly looking for feedback around the foldout format, and if we disregard the comment by my fellow writer of AI risk explanations, the comments were all about the ideas instead of the formatting. 

[^k0dbkuzxv8]: I did botch the marketing of my post, but the fact that the AISafety.info folks did a better job of their asks and still got minimal traction makes me think that improving my marketing would not have helped. 

[^fl9pfizlrz]: Directly requesting feedback from specific people obviously works, and in my experience requesting feedback from a local mailing list will garner feedback as well, but posting to a general social media-esque site doesn’t. It seems obvious that people will gravitate towards discussing new ideas, not towards the grungy work of giving feedback, and I can hardly begrudge them for that. 

[^bkw8tfgi4b]: One of the AISafety.info authors did drop a comment on my post; unlike me, they understood the value of solidarity. 

[^kzd1qzfq7u]: On the other hand, taking only a month break may have worked if the project scope was smaller: I did write fairly regularly for 2 years, and produced a lot of text. It just wasn't enough for the scale of the project. 

[^9qwk8fyw8ju]: This example of a heavily folded/outlined article which is nonetheless popular might be a counterexample; perhaps it points to the format being suitable for references, and not general arguments? Perhaps it just did better karma-wise because it wasn't asking for feedback?