by kypro | 2025-06-23 23:57:06+00:00
No – I like this. Most people who talk about AI risk, do not take it seriously in my opinion. I've been having nightmares about AIs since I was kid and these have only become more frequent and more realistic the older I've gotten. I studied AI at university because I was always scared and equally fascinated by AI. Even today most of my friends are friends because they're the only people I've been able to find who are willing to talk to me about AI risk for hours on end. Everyone else thinks I'm crazy, but I take this super seriously and I get frustrated by people who treat it purely as a critical thinking exercise, given in all likelihood their family and children will be dead or worse in a few years.

To answer your question at very a high-level, I'm pretty depressed so I'm not bothered at all about dying which helps with the emotional side of this. I'm not bothered that I think I'll be dead soon – and a quick death is my highest probability positive outcome for what is coming for me personally.

My primary concern, and the thing that keeps me up at night, is the risk of new forms of torture which are so inconceivably bad that any positive outcomes from ASI would never be worth the risk. In a few years if things go "well" and we get ASI it should be completely feasible to 3d print a torture helmet which will simulate the feeling of being burnt to death for 100,000+ years. Assuming this is an experience that's physically possible to experience then signalling the brain to experience it is just an engineering problem – and likely a fairly trivial one for an ASI. Again, death should be seen as a very positive outcome. ASI will create hell in the literal sense. The question here isn't whether ASI could create hells, or whether someone out there might be messed up enough to send someone to hell, but whether humanity will create ASI. If you think we will then we will create hell and we will banish people to it.

So if there's even a 0.1% chance of something this bad happening to me I plan to check out early. That said, I see this as a low-probability risk on an individual basis – I think it's much much more likely I'll die of a designer virus or something similar. But should this not happen then we should assume many people alive today will be subjected to unimaginable horrors. Use cases of new forms of torture are very obvious and will be rapidly adopted in totalitarian states, but I'd also question whether democracies will end up as totalitarian states in a world where ASI exists – I think this depends on whether you believe ASI is likely to concentrate power (which I suspect it probably will).

I appreciate what I'm saying sounds crazy today, but magical sticks which go "bang" and immediately kill the people they're pointed towards also sounds crazy to most humans who ever lived. We always underestimate how much technology can alter the boundaries of fiction, and ASI will likely change those boundaries faster and more significantly than we can possibly imagine today. Even I as a AI doomer am probably understating the extent of the horror which could be coming to you and your family. Your kids may suffer unimaginably for millions of years, trillions even.

More likely though is that ASI introduces several new civilisational risks in rapid succession and one or two of them kill most humans alive today. I'd consider this a mostly neutral outcome.

Other low probability, slightly bad outcomes include scenarios where AI advancements are fundamentally destabilising and where events like mass-job loss, rioting and war results in a halting of civilisational progress and mass death – largely from famine. I'm preparing for these outcomes as much as I possibly can since this these are the only outcomes I have any agency over. I grow a lot of food, I have chickens and keep years of food supplies (needed for nuclear winter scenarios). I'm in the process of fitting a water butt in my garden so I'm not dependant on the public water supply. I keep a large sash of firewood and heating fuel for cooking and heat if energy is cut off. I have been amassing tools for a few years now so I can repair things and can produce various items (including weapons). I will continue preparing for these scenarios for as long as I have left.

Realistically though there's very little I can do in most scenarios and I'm no where near as prepared as I'd like to be. Really I just hope to avoid suffering and must accept most outcomes I have very little agency over. I just hope to die quickly or have the strength to check out while I have the chance.

I guess the sad truth is I'd consider being diagnosed with terminal cancer right now as a positive improvement in my expected life outcome. It's quite hard to over state how concerned I am. The end of humanity isn't the bad outcome imo, it's neutral from the perspective of human suffering and dramatically understates the risks coming.

But hey, hopefully I'm wrong =)