Harley-ml/lesswrong huggingface dataset:
# Is a random box of gas predictable after 20 seconds?

This came up as a tangent from [this question](https://www.lesswrong.com/posts/HT8SZAykDWwwqPrZn/will-quantum-randomness-affect-the-2028-election), which is itself a tangent from a discussion on [The Hidden Complexity of Wishes](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/).

Suppose we have a perfect cubical box of length 1 meter containing exactly 1 mol of argon gas at room temperature.

*   At t=0, the gas is initialized with random positions and velocities $\theta$ drawn from the Maxwell-Boltzmann distribution.
*   Right after t=0 we perturb one of the particles by 1 angstrom in a random direction $X$ to get the state $m(\theta, X)$.
*   All collisions are perfectly elastic, so there is no viscosity \[edit, this is wrong; even ideal gases have viscosity\] and energy is conserved.
*   For each possible perturbation, we run physics forward for 20 seconds and measure whether there are more gas molecules in the left side or right side of the box at t=20 seconds (the number on each side will be extremely close to equal, but differ slightly). Do more than 51% of the possible perturbations result in the same answer? That is, if $is\_left$ is the predicate "more gas molecules on the left at t=20", is $|\mathop{Pr}_{X}[is\_left(m(\theta, X))] - 0.5| > 0.01$?

This is equivalent to asking if an omniscient forecaster who knows the position and velocity of all atoms at t=0 except for 1 angstrom of uncertainty in 1 atom can know with >51% confidence which side has more gas molecules at t=20.

I think the answer is no, because multiple billiard balls is a textbook example of a chaotic system that maximizes entropy quickly, and there's no reason information should be preserved for 20 seconds. This is enough time for each atom to collide with others millions of times, and even sound waves will travel thousands of meters and have lots of time to dissipate.

[@habryka](https://www.lesswrong.com/users/habryka4?mention=user) thinks the answer is yes and the forecaster could get more than 99.999% accuracy, because with such a large number of molecules, there should be *some* structure that remains predictable.

Who is right?
# Importing a Python File by Name

Let's say you have a python script:

2024-01-24--evaluate-chimeras.py

And you want to pull a section of it out into a separate file:

2024-01-25--strand-split-artifacts.py

You would hope you could just do something like:

import "/path/to/2024-01-25--strand-split-artifacts.py"

But this doesn't work: `import` wants a module name, not a filename. The simplest way I know to import a python file from a path is:

import sys
import importlib

sys.path.append("/path/to/")
ssa = importlib.import_module("2024-01-25--strand-split-artifacts")

There are a lot of [complicated ways to do this](https://stackoverflow.com/questions/67631/how-can-i-import-a-module-dynamically-given-the-full-path), some of which avoid needing to add something to `sys.path`, but for quick one-off research code better to keep it simple.
# "Does your paradigm beget new, good, paradigms?"

*A very short version of this post, which seemed worth rattling of quickly for now.*

A few months ago, I was talking to John about paradimicity in AI alignment. John says "we don't currently have a good paradigm." I asked "Is 'Natural Abstraction' a good paradigm?". He said "No, but I think it's something that's likely to output a paradigm that's *closer* to the right paradigm for AI Alignment."

"How many paradigms are we away from the right paradigm?"

"Like, I dunno, maybe 3?" said he.

Awhile later I saw John arguing on LessWrong with (I think?) Ryan Greenblatt about whether Ryan's current pseudo-paradigm was good. (Sorry if I got the names here or substance here wrong, I couldn't find the original thread, and it seemed slightly better to be specific so we could dig into a concrete example).

One distinction in the discussion seemed to be something like:

*   On one hand, Ryan thought his current paradigm (this might have been "[AI Control](https://www.lesswrong.com/posts/kcKrE9mzEHrdqtDpE/the-case-for-ensuring-that-powerful-ais-are-controlled)", as contrasted with "AI Alignment") had a bunch of traction on producing a plan that would at least reasonably help if we had to align superintelligent AIs in the near future.
*   On the other hand, John argued that the paradigm didn't feel like the sort of thing that was likely to bear the fruit of *new, better paradigms.* It focused on an area of the superintelligence problem that, while locally tractable, John thought was insufficient to actually solve the problem, and also wasn't the sort of thing likely to pave the way to new paradigms.

Now a) again I'm not sure I'm remembering this conversation right, b) whether either of those points are true in this particular case would be up for debate and I'm not arguing they're true. (also, regardless, I am interested in the idea of AI Control and think that getting AI companies to actually [do the steps necessary to control at least nearterm AIs](https://www.lesswrong.com/posts/thkAtqoQwN6DtaiGT/carefully-bootstrapped-alignment-is-organizationally-hard) is something worth putting effort into)

But it seemed good to promote to attention the idea that: when you're looking at clusters of AI Safety research and thinking about whether it is congealing into a useful, promising paradigm, one of the questions to ask is not just "does this paradigm seem locally tractable" but "do I have a sense that this paradigm will open up new lines of research that can lead to be better paradigms?".

(Whether one can be accurate in answering that question is yet another uncertainty. But, I think if you ask yourself "is this approach/paradigm *useful"*, your brain will respond with different intuitions than "does this approach/paradigm seem likely to result in new/better paradigms?")

Some prior reading:

*   [Look For Principles Which Will Carry Over To The Next Paradigm](https://www.lesswrong.com/posts/bmoQ2wy7Nd7EiJdpg/look-for-principles-which-will-carry-over-to-the-next) 
*   [Open Problems Create Paradigms](https://www.lesswrong.com/posts/whXFkYEDiDk8tSBqH/open-problems-create-paradigms)
# RAND report finds no effect of current LLMs on viability of bioterrorism attacks

Key Findings
------------

*   This research involving multiple LLMs indicates that biological weapon attack planning currently lies beyond the capability frontier of LLMs as assistive tools. The authors found no statistically significant difference in the viability of plans generated with or without LLM assistance.
*   This research did not measure the distance between the existing LLM capability frontier and the knowledge needed for biological weapon attack planning. Given the rapid evolution of AI, it is prudent to monitor future developments in LLM technology and the potential risks associated with its application to biological weapon attack planning.
*   Although the authors identified what they term *unfortunate outputs* from LLMs (in the form of problematic responses to prompts), these outputs generally mirror information readily available on the internet, suggesting that LLMs do not substantially increase the risks associated with biological weapon attack planning.
*   To enhance possible future research, the authors would aim to increase the sensitivity of these tests by expanding the number of LLMs tested, involving more researchers, and removing unhelpful sources of variability in the testing process. Those efforts will help ensure a more accurate assessment of potential risks and offer a proactive way to manage the evolving measure-countermeasure dynamic.

The linkpost is to the actual report, see also their [press release](https://www.rand.org/news/press/2024/01/25.html).
# Workshop (hackathon, residence program, etc.) about for-profit AI Safety projects?

Reading posts like "[This might be the last AI Safety Camp](https://www.lesswrong.com/posts/EAZjXKNN2vgoJGF9Y/this-might-be-the-last-ai-safety-camp)" makes me feel very sad. The degree to which AI Safety is currently funding-constrained is clearly inadequate on the national and civilisational levels.

Maybe let's try to make a smart counter-move and accelerate the development of [for-profit AI Safety projects](https://www.lesswrong.com/posts/PcTLHamp236afJxxT/some-for-profit-ai-alignment-org-ideas) (see also the comments to that post, and [this post](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment))? With the obvious idea to pull some VC money, which is a different pool than AI safety philanthropic funds.

Potential collaborations:

*   [https://vc.ae.studio/](https://vc.ae.studio/)
*   [https://www.lionheart.vc/](https://www.lionheart.vc/)
*   [https://www.mythos.vc/](https://www.mythos.vc/)
*   [https://www.joinef.com/](https://www.joinef.com/) (or maybe they are working on something like this already? At least [Matt Clifford](https://twitter.com/matthewclifford) was the UK PM's sherpa for AI Safety Summit, and now working on AI full time)
*   More?

P.S. I'm not a professional organiser or community builder, nor a startup accelerator program manager, so just floating the idea, but I'd be very eager to participate if something like this is organised.
# Stacked Laptop Monitor

A few weeks ago I [broke my laptop's screen](https://www.jefftk.com/p/reviving-a-2015-macbook) and was speculating on whether [putting a portable monitor above](https://www.jefftk.com/p/screen-supported-portable-monitor) the screen would do what I wanted. The hope was that it would let me set up a laptop where both my wrists and neck were in comfortable positions. It does!

[![](https://www.jefftk.com/stacked-portable-monitor.jpg)](https://www.jefftk.com/stacked-portable-monitor-big.jpg)

The monitor is a [1080p model](https://www.amazon.com/Portable-Monitor-Upgraded-1920x1080-Computer/dp/B07ZLY26FW/) that gets it's power and video signal over USB-C. It's screwed to an aluminum plate with M4 VESA screws, which is connected to another plate with [butler tray hinges](https://www.amazon.com/dp/B09N6FTHN6):

[![](https://www.jefftk.com/stacked-portable-monitor-mechanism.jpg)](https://www.jefftk.com/stacked-portable-monitor-mechanism-big.jpg)

Then the bottom plate velcros to the back of the laptop screen:

[![](https://www.jefftk.com/stacked-portable-monitor-velcro.jpg)](https://www.jefftk.com/stacked-portable-monitor-velcro-big.jpg)

I probably should have gotten some kind of hinge that locks, but since I didn't I cut a piece of wood to chock it:

[![](https://www.jefftk.com/stacked-portable-monitor-chock.jpg)](https://www.jefftk.com/stacked-portable-monitor-chock-big.jpg)

I should sort out something sturdier and harder to lose.

It folds up, with the portable monitor's cover protecting the screen:

[![](https://www.jefftk.com/stacked-portable-monitor-folded.jpg)](https://www.jefftk.com/stacked-portable-monitor-folded-big.jpg)

It still fits in my backpack's laptop pocket, but only just.

I tried it on the airplane yesterday, and it's so much better ergonomically!

[![](https://www.jefftk.com/stacked-portable-monitor-on-airplane.jpg)](https://www.jefftk.com/stacked-portable-monitor-on-airplane-big.jpg)

It also doesn't need the chock, since to fold up the seat in front of me would need to not be there.

There are a few downsides, though:

*   I need to be careful to position it so all the forces are aligned, and only use it in places where it's not going to get jostled. Building a kickstand (which all the others seem to have) would probably help a lot with this.
    
*   If you use it in a meeting you can't see the person across from you:
    
    [![](https://www.jefftk.com/stacked-portable-monitor-meeting.jpg)](https://www.jefftk.com/stacked-portable-monitor-meeting-big.jpg)
    
*   If you're nine it's probably too high for you:
    
    [![](https://www.jefftk.com/stacked-portable-monitor-nine.jpg)](https://www.jefftk.com/stacked-portable-monitor-nine-big.jpg)
    

I've only used it a bit, but so far I'm very happy with it. I'm composing this post on it while traveling, and it's way more comfortable.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0ZU5UmLNQeDxdVVFv344xnt1VCMZ5QxNMXGFwkqtpzkWJAgXqiX9vXWhgx1PXTbZKl), [mastodon](https://mastodon.mit.edu/@jefftk/111822621398712140)*
# Aligned AI is dual use technology

Humans are mostly selfish most of the time. Yes, many of us dislike hurting others, are reliable friends and trading partners, and care genuinely about those we have personal relationships with. Despite this, spontaneous strategic altruism towards strangers is extremely rare. The median American directs exactly $0 to global poverty interventions, and that is a true statement regardless of whether you limit it to the Americans that make ten, fifty, a hundred, or a thousand times as much money as Nigerians.

Some people hope that with enough tech development we will reach a "post-scarcity" regime where people have so much money that there is a global commons of resources people can access largely to their hearts' content. But this has always sounded to me like a 1023 AD peasant hoping that in 2023, the French will be so rich that no one outside France will die of a preventable disease. There will always be more for people with money to consume; even in the limits of global wealth, the free energy or resources that a person could devote to helping poor people or defending them from abuse could also be devoted to extending a personal lifespan before heat death.

So in keeping with this long tradition of human selfishness, it sounds likely that if we succeed at aligning AI, the vast, vast majority of its output will get directed toward satisfying the preferences and values of the people controlling it (or possessing leverage over its continued operation) - not the "CEV of all humans", let alone the "CEV of all extant moral persons". A person deciding to use their GPUs to optimize for humanity's betterment would be the equivalent of a person hiring a maid for humanity instead of their own home; it's simply not what you expect people to do in practice, effective altruists aside. In a "polytheistic" future where at least a dozen people share large amounts of control, I expect wielding this control will involve:

*   Extracting any significant extant resources from the remainder of people vulnerable to manipulation or coercion.
*   Creating new people of moral value to serve as romantic partners, friends, and social subordinates.
*   Getting admiration, prestige, and respect from legacy humans, possibly to extreme degrees, possibly in ways we would dislike upon reflection.
*   Engineering new worlds where they can "help" or "save" others, depending on the operational details of their ethics.

In this scenario the vast majority of beings of moral worth spread across the galaxy are not the people the AIs are working to help. They're the things that *surround* those people, because those oligarchs enjoy their company. And it doesn't take a genius to see why that might be worse overall than just paperclipping this corner of the cosmos, depending on who's in charge and what their preferences for "company" are, how they react to extreme power, or how much they care about the internal psychology of their peers.
# Virtually Rational - VRChat Meetup

![Image](https://media.discordapp.net/attachments/1200940507719815300/1200940964605333524/VRChat_2024-01-27_22-13-48.800_1920x1080.png?ex=65c8026e&is=65b58d6e&hm=5281d54110dd91ca7c93b0ec5803b85320fd6d8c6cae35bed35f51282a84f5ad&=&format=webp&quality=lossless&width=1836&height=1034)

The next meetup will be held in this VRChat world:[http://tinyurl.com/4df87mt4](http://tinyurl.com/4df87mt4). Or search for "LESSWR.7138" under groups if you have trouble with the link.

Please RSVP above.

And don't forget to join our discord: [https://discord.gg/U4GZSGmaF2](https://discord.gg/U4GZSGmaF2)
# Palworld development blog post

Palworld [is currently](https://steamcharts.com/top) the most-played game on Steam. It's by a small Japanese company called [Pocketpair](https://store.steampowered.com/developer/pocketpair).

Shortly before Palworld released, the CEO put up [this blog post](https://note.com/pocketpair/n/n54f674cccc40); [here's a translation](https://note-com.translate.goog/pocketpair/n/n54f674cccc40?_x_tr_sl=ja&_x_tr_tl=en). Here are some points I thought were interesting:

- on the co-founder:

> It took me three years to quit JP Morgan, where I joined as a new graduate. He quit after only a month. The more talented people are, the sooner they leave the company.

- on one of the animators:

Looking for someone in Japan who had experience with guns in games, he looked on twitter and found someone posting gun reloading animations. Contacting this person, it turned out they were a 20-year-old high school dropout working part-time at a convenience store in Hokkaido. Pocketpair hired him as a remote employee for a month, then asked him to come to Tokyo. His parents thought it must definitely be some kind of scam, but he went, and did a lot of different animation work, and was a very effective employee.

- on the main character designer:

She was rejected during the initial resume screening. A few months later, they tried recruiting again, she DM'd him on twitter, and ended up being hired. In the meantime, she'd applied to about 100 companies and was rejected by all of them.

> And she now draws most of the characters in Palworld. She is a new graduate, and she applied to nearly 100 companies but was rejected by all of them. (...) She doesn't like to use the word genius, but she might be a genius.

---

I thought that post indicated some interesting things about typical hiring processes, credential evaluation, and how effectively society is utilizing talent.

Typically, people say that the market is mostly efficient, and if there was financial alpha to be gained by doing hiring differently from most corporations, then there would already be companies outcompeting others by doing that. Well, here's a company doing some things differently and outcompeting other companies. Maybe there aren't enough people willing to do such things (who have the resources to) for the returns to reach an equilibrium?
# What exactly did that great AI future involve again?

I've been trying my best to think of something that AGI could do which I really love and deeply endorse.

I can think of some nice things. New kinds of animals. Non-habit-forming heroine. Stupid-pointless-disagreements-between-family-members fixomatic maybe. Turn everyone hot. 

None of this makes me joyful and hopeful. Just sounds neat and good. Humans seem pretty damn good at inventing tech etc ourselves anyway.

I think I might have assumed or copy-pasted the "AI is truly wonderful if it goes truly well" pillar of my worldview. Or maybe I've forgotten the original reasons I believed it.

What exactly did that great AI future involve again?
# Things You're Allowed to Do: At the Dentist

*Inspired by Milan Cvitkovic’s article,* [*Things You’re Allowed to Do*](https://milan.cvitkovic.net/writing/things_youre_allowed_to_do/)*.*

Going to the dentist can be uncomfortable. Some amount of this is unavoidable.

Yet most dentists and staff care a lot about patient comfort. Tell them what you need, and you may very well get it!

The hardest part is figuring out what’s on the menu. Below are some items that I’ve discovered.

Caveats:

*   available options may vary a lot by dentist
*   options sometimes have tradeoffs, which you should discuss with your dentist

You can control the suction
---------------------------

Every time I go to the dentist, there is a segment featuring a water hose, a suction hose, and a third or fourth bonus tool in my mouth.

I find this uncomfortable for many reasons:

*   the suction hose is badly positioned and water is accumulating
*   the suction hose hits the back of my throat and I gag and cough
*   I am nervously anticipating any of the above

Luckily, if I ask, I can hold the suction hose myself. I can position it exactly as needed for my comfort.

Dentists seem to like this too since it frees up one of their hands.

You can get smaller x-ray films
-------------------------------

I have a small jaw and find the bitewing x-rays to be super large and uncomfortable. Sometimes they make me gag, and that usually means I am more likely to gag on the next try. I don’t like it.

It turns out that my dentist has smaller bitewings on hand. They are designed for children but work for adults too, and I find them to be much more comfortable.

The main downside is that they might make it a bit harder for the dentist to get the specific images they want.

You can refuse polish
---------------------

I don’t like the feeling of having my teeth polished, and often the sickly artificial flavour gives me a headache afterward*.

Usually the stains on my teeth are mild and have already been removed during scaling**.

So do I really need to have my teeth polished?

\* *I find that flavourless polish also helps here.*

\*\* *Some people loathe scaling and tolerate polishing. Maybe you can trade more of one for less of the other?*

Misc. roundup
-------------

*   ask for different painkiller options to get something more personally effective or less aversive (e.g. needles)
*   decline painkillers to save time during mild procedures
*   ask for water or a tissue at any time
*   ask to pause for a minute
*   decline the crappy free toothbrush they give you at the end
*   ask for a free brushhead that works with the electric toothbrush you use at home
# Has anyone actually changed their mind regarding Sleeping Beauty problem?

[https://en.wikipedia.org/wiki/Sleeping\_Beauty\_problem](https://en.wikipedia.org/wiki/Sleeping_Beauty_problem)

If that happened what was the argument that did it for you?

I'm interested in situations where a person used to think that the correct answer was 1/2 and then, on a reflection, decided that it's actually 1/3 , or vice versa, not when the resulting belief is that the question is meaningless or both answers are valid.
# Clip keys together with tiny carabiners

I've always found it annoying to have a bunch of keys in my pocket (they're heavy, sometimes poke into you, and create weird bulges), so I try to carry as few as possible. Unfortunately, I've reached a point where I have a fairly large number of keys that need to keep with me sometimes:

*   Default key chain: Apartment key, apartment key fob, USB security key, mail key
*   Car key
*   Bike key
*   Storage keys
*   Multitool (I know, it's weird that I don't always carry this, but it's heavy)

![A bunch of keys](https://www.brendanlong.com/images/keyring-carabiners/many-keys.webp)

Previously I kept all of these except the storage keys on one keyring, but I recently realized that tiny carabiners exist, which means I can keep all of the non-default keys clipped to a storage area and then add just the keys I need during the day.

![Keys on carabiners lined up next to a smaller key ring](https://www.brendanlong.com/images/keyring-carabiners/keys-organized.webp)

For example, most days I just need either my car key or bike key.

![A small key ring with a car key carabinered to it](https://www.brendanlong.com/images/keyring-carabiners/smaller-keyring.webp)

The downside of this is that the caribiner itself adds size and weight, so if I needed to carry the full set of keys it would be even worse. Luckily, that doesn't seem to happen to me (and some keys are mutually-exclusive, like the bike and car key). If you do have a large number of keys you need together, this won't help you.

The specific ones I bought were [this 12-pack of tiny carabiners on Amazon](https://www.amazon.com/dp/B0B6HZW67D), but I don't have any particular faith in this clearly-random-name-generator company. This size is perfect for the non-hook size to hold a tiny key ring with 2-3 keys. I didn't get double-carabiners because I don't have any need to quickly detach the keys themselves, and that would make them slightly bigger.

I thought this might be a place where other people who care unusually-much about the exact weight of their pockets might congregate, so if you feel similarly, consider getting some tiny keyring carabiners.
# Without Fundamental Advances, Rebellion and Coup d'État are the Inevitable Outcomes of Dictators & Monarchs Trying to Control Large, Capable Countries

  
A pdf version of this report will be available

Summary
=======

In this report, I argue that dictators ruling over large and capable countries are likely to face insurmountable challenges, leading to inevitable rebellion and coup d'état. I assert this is the default outcome, even with significant countermeasures, given the current trajectory of power dynamics and governance, and therefore when we check real-world countries we should find no or very few dictatorships, no or very few absolute monarchies and no arrangements where one person or a small group imposes their will on a country. This finding is robust to the time period we look in - modern, medieval, or ancient.   
  
In Section 1, I discuss the countries which are the focus of this report. I  am specifically focusing on nations of immense influence and power (with at least 1000 times the Dunbar Number of humans) which are capable of running large, specialized industries and fielding armies of at least thousands of troops. 

In Section 2, I argue that subsystems of powerful nations will be approximately consequentialist; their behavior will be well described as taking actions to achieve an outcome. This is because the task of running complex industrial, social and military systems is inherently outcome-oriented, and thus the nation must be robust to new challenges to achieve these outcomes.

In Section 3, I argue that a powerful nation will necessarily face new circumstances, both in terms of facts and skills. This means that capabilities will change over time, which is a source of dangerous power shifts.

In Section 4, I further argue that governance methods based on fear and suppression, which are how dictatorships might be maintained, are an extremely imprecise way to secure loyalty. This is because there are many degrees of freedom in loyalty that aren't pinned down by fear or suppression. Nations created this way will, by default, face unintended rebellion.

In Section 5, I discuss why I expect control and oversight of powerful nations to be difficult. It will be challenging to safely extract beneficial behavior from misaligned groups and organizations while ensuring they don’t take unwanted actions, and therefore we don’t expect dictatorships to be both stable and aligned to the goals of the dictator

Finally, in Section 6, I discuss the consequences of a leader attempting to rule a powerful nation with improperly specified governance strategies. Such a leader could likely face containment problems given realistic levels of loyalty, and then face outcomes in the nation that would be catastrophic for their power. It seems very unlikely that these outcomes would be compatible with dictator survival.  
  
*\[\[Work in progress - I'll add to this section-by-section\]\]*  
  
*Related work  -* [*https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe*](https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe)
# Per protocol analysis as medical malpractice

“Per protocol analysis” is when medical trial researchers drop people who didn’t follow the treatment protocol. It is outrageous and must be stopped.

For example: let’s say you want to know the impact of daily jogs on happiness. You randomly instruct 80 people to either jog daily or to simply continue their regular routine. As a per protocol analyst, you drop the many treated people who did not go jogging. You keep the whole control group because it wasn’t as hard for them to follow instructions.

At this point, your experiment is ruined. You ended up with lopsided groups: people able to jog and the unfiltered control group. It would not be surprising if the eager joggers had higher happiness. This is confounded: it could be due to preexisting factors that made them more able to jog in the first place, like being healthy. You’ve thrown away the random variation that makes experiments useful in the first place.

This sounds ridiculous enough that in many fields per protocol analysis has been abandoned. But…not all fields. Enter the [Harvard hot yoga study](https://www.health.harvard.edu/blog/yoga-could-complement-traditional-treatment-for-depression-2017090712369), studying the effect of hot yoga on depression.

![](https://lh7-us.googleusercontent.com/zgFGvZ_tplREQIrMe4dS2L8D5Hh1tsnY5uBWbUbpG8UJIolT_N1D_omx8LfxS6r3yreNMvRp9xBAYuDu6r643L--UwahXY-xhOt6_T78G23lafpI038NhfdXSpD2KvB9Ys5oZMG4Y1bLm6QDaTaaxPY)

If the jogging example sounded contrived, this study actually did the same thing but with hot yoga. The treatment group was randomly assigned to do hot yoga. Only 64% (21 of 33) of the treatment group remained in the study until the endpoint at the 8th week compared to 94% (30 of 32) of the control. They end up with striking graphs like this which could be *entirely* due to the selective dropping of treatment group subjects.

![](https://lh7-us.googleusercontent.com/P43Wv3no5qiH0B5-7QK2pHQiCH-vY3ah8ByxNbRD5H6ojxY-2olpkjGtBLUczIiNRPTejfWtUwua_-Z-PvmYCGtr6O03jrm-6yOikER1gopE-brQnpGSgtdDi2_gXjBpfTmTnOB8Xe0Cvb-dUSspg1w)

What’s depressing is that there is a known fix for this: intent-to-treat analysis. It looks at effects based on the original assignment, regardless of whether someone complied or not. The core principle is that every comparison should be split on the original random assignment, otherwise you risk confounding. It should be standard practice to report the intent-to-treat and many medical papers do so---at least in the appendix somewhere. The hot yoga study does not.

It might be hard to estimate this if you’re following people over time and there’s a risk of differential attrition---you're missing data for a selected chunk of people. 

Also, hot yoga could still really work! But we just don’t know from this study. And with all the buzz, there’s a good chance this paper ends up being worse than useless: leading to scaled-up trials with null findings that might’ve not been run if there had been more transparency to begin with.
# PIBBSS Speaker events comings up in February

*TLDR: The PIBBSS Speaker Series is happening in mid-February. Read more about the talks and register *[*on our website*](https://pibbss.ai/speaker-series/)*. Sign up for future updates about our speaker events* [*here*](https://forms.gle/6RjCBUWSsZzHbDiV7)*.*

* * *

[PIBBSS](http://pibbss.ai)  regularly invites researchers from a wide range of fields studying intelligent behaviour and exploring their connections to the problem of AI Risk and Safety.

In the past, we had speakers such as **Michael Levin** talking about [Morphogenesis as a form of intelligent problem-solving building towards a framework of Basal Cognition](https://www.youtube.com/watch?v=RwEKg5cjkKQ&list=PL0oAgcyKrxGsVkKX-UGXFNiHOnPDN80K5&index=3); **Josh Bongard** about [efforts to build hybrid intelligent 'agents' at the intersection of biology, cognition and engineering;](https://www.youtube.com/watch?v=7-wvArSvHsc&list=PL0oAgcyKrxGutxkXQJK73gRvD-fXdlfAp&index=7) **Miguel Aguilera** on  [Nonequilibrium Neural Computation](https://www.youtube.com/watch?v=b762Y9wU5IM&list=PL0oAgcyKrxGutxkXQJK73gRvD-fXdlfAp&index=6); **Steve Byrnes on** [Challenges for Safe and Beneficial Brain-Like AI](https://www.youtube.com/watch?v=x7L6fC8z8XU&list=PL0oAgcyKrxGutxkXQJK73gRvD-fXdlfAp&index=3); or **Tang Zhi Xuan** on [Modeling Bayesian Bounded Rational Agents](https://www.youtube.com/watch?v=mym_r4eM7tc&list=PL0oAgcyKrxGsVkKX-UGXFNiHOnPDN80K5&index=2). 

We have a couple of speaker events are coming up in February.

*   Konrad Körding on "Interpretability of brains and neural networks"
    *   **Feb 13**,9AM PST/5PM UTC **(**[**Register**](https://us06web.zoom.us/webinar/register/WN__x_VXSeiQ7KZPnSOo8LyNQ)**)**
*   Daniel Polani on "Information and its Flow: a Pathway towards Intelligence"
    *   **Feb 16**,9AM PST/5PM UTC, **(**[**Register**](https://us06web.zoom.us/webinar/register/WN_USVj1bDJQvG0NMQzDjm1lA)**)**
*   Nathaniel Virgo on "Mathematical Perspectives on Goals and Beliefs in the Physical World: Extending on the 'Good Regulator Theorem'"
    *   **Feb 19**, 6PM PST,/20th 2AM UTC, **(**[**Register**](https://us06web.zoom.us/meeting/register/tZcrdOCuqzwjGdUgFGYaELXDrud-LuqlsNcw)**)**
*   Tsvi Benson-Tilsen on “Creating the contexts needed to produce the concepts needed to understand minds” 
    *   **20 Feb **9AM PST/5PM UTC **(**[**Register**](https://us06web.zoom.us/webinar/register/WN_mWRwFsHCRTqkhGtb3t98MQ#/registration)**)**
*   Abram Demski on “Meaning & Agency" 
    *   **23 Feb **9AM PST, 5PM UTC, **(**[**Register**](https://us06web.zoom.us/meeting/register/tZYocOGpqjsjEtcR9-dnsRKz8uqMjjbWwI0k)**)**

If you want to be informed about future events (we typically won't post about this going forward), signup [here](https://forms.gle/6RjCBUWSsZzHbDiV7) to our speaker event specific email list.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9c3f9aa048d1148c7a7aa2b08afd814ea6196f88448e62d4.png)

*Snapshots from past talks*
# Wrong answer bias

The correct answer leaves nothing to be said. The wrong answer starts a conversation, a research program, an investigation, a journey, an institute, a paper, a book, a youtube channel, a lifestyle, a tribe.

Why is reading a textbook so boring? Very few people do it. It just has right answers and few great questions to ponder. Blogs arguing about nutrition are great reading though.

If you write a research paper saying "all that shit is dumb just do the obvious thing" then you'll probably have trouble getting it published. I tried once but my professor shut it down saying it's a bad strat basically.

"Clean energy" is great as a conversation piece or phrase in your mission statement but "just deregulate nuclear lmao why are you wasting your time" is definitely not in my experience.

**It's fine if bad ideas are all we ever talk about but the trouble comes when it's time for someone to sit down and do their work.** The doctor trying to cure a thing mostly heard about the RCTs on all the shitty methods that barely do anything, so they pick their favorite of those. The AI safety implementer mostly heard discussions about bad methods and probably tries to patch one of those. (And fixing the mistake in a bad method is a not a good way to make a good method.)  The parents in the parent group of course mostly talk about their failed attempts to fix the problems they do have (and mostly forget about the problems they never had or quickly fixed) so the parents go home and try the mediocre ideas.

Have hope, you can combat wrong answer bias with these simple tricks.

*   Do not write or speak
*   Do not read or listen
*   Hate the process of fixing things
*   Savor the experience of shit working like it should
*   Act like you have no time or money and won't ever have any (helps if you don't and won't)
*   Assume your answer is too complicated
*   Assume your question is too loaded
*   Assume any task is actually really easy you're doing it wrong
*   Hate struggle
*   Hate nuance
*   If you look impressive then you're doing it wrong
*   Love true success
*   If it looks like you barely did anything at all then you're doing it right
*   Observe people & orgs who get shit done or don't have a problem you have
*   Turn your back on anyone struggling with anything for any reason. They're probably looking for a frend or comunity instead of  just focusing on nailing shit all the time like you.
# Davidad's Provably Safe AI Architecture - ARIA's Programme Thesis

The programme thesis of Davidad's agenda to develop provably safe AI has just been published. For context, [Davidad](https://www.aria.org.uk/what-were-working-on/#davidad) is a Programme Director at [ARIA](https://www.aria.org.uk/) who will grant somewhere between £10M and £50M over the next 3 years to pursue his research agenda. 

It is the most comprehensive public document detailing his agenda to date.

Here's the most self-sufficient graph explaining it at a high-level although you'll have to dive into the details and read it several times to start grasping the many dimensions of it. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/70a83c7124615a97239007e05759e06e0a08e33a967e147c.png)

I'm personally very excited by Davidad's moonshot that I currently see as the most credible alternative to scaled transformers, which I consider to be too flawed to be a credible safe path, mostly because:

*   Ambitious LLM interpretability seems very unlikely to work out: 
    *   Why: the failed attempts at making meaningful progress of the past few years + the systematic wall of understanding of ~80% of what's going on across reverse engineering attempts 
*   Adversarial robustness to jailbreak seems unlikely to work out:
    *   Why: failed attempts at solving it + a theoretical paper of early 2023 that I can't find right now + increasingly large context windows
*   Safe generalization with very high confidence seems quite unlikely to work out
    *   Why: absence of theory on transformers + weak interpretability

A key motivation for pursuing moonshots a la Davidad is, as he explains in his thesis, to shift the incentives from the current race to the bottom, by derisking credible paths to AI systems where we have strong reasons to expect confidence in the safety of systems. See the graph below: 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/77f3f0768f12b5f837b99480d39f1b26c53da36412164df3.png)
# Running a Prediction Market Mafia Game

In a normal game of mafia/werewolf, there are a few high-level strategic elements:

*   **Night Play**: During night phases, the mafia has to choose who to try to kill and the players with night actions (cops, doctors, and in some cases trackers, watchers, vigilantes, etc.) have to choose who to target.
*   **Day Play:** During day phases, the townies try to share information without exposing too much information about the identities of their power roles (cops, doctors, etc.) while the mafia tries to provide misleading information. Everyone is trying to affect (1) future night actions and (2) the daily execution vote.

The goal of the prediction market mafia game is to replace the voting mechanism with a set of markets, one for each living player on whether they’re a member of the mafia. At the end of each day phase, the player with the highest market price is executed. 

If you replace the election with a market that the *players* participate in, the game will break because townies have an incentive to bet their own markets down to 0% and to pressure mafia players to do the same and drain their capital (I didn’t think about this that hard; maybe a few adjustments actually suffice to make this setup work, but I don’t think so). 

A simple way to make it work is to let the *spectators* bet on the prediction markets. So now there are multiple parties to the game: 

*   the town players, trying to kill all of the mafia players;
*   the mafia players, trying to kill the town players; and
*   the speculators, trying to maximize their profits.

I used the term “spectators” above, but the speculators need to have everyone’s night actions hidden from them the same way players do.

Tweaks/considerations:

*   The end of the day phase could be decided by some kind of random process that avoids the problem of rapid trading activity right before the phase ends.
*   There can be other speculative markets that aren’t execution-relevant, such as on the identity of particular players or roles, the number of phases the game will take, the eventual winning side, and so on, but this might lead to weird outcomes where speculators move the execution-relevant markets with the intent of manipulative other markets, such as by trying to take a loss by getting a town player executed because this will increase profits on other markets.
*   In most setups, most dead players won’t be able to join the speculators, because they’ll possess game-breaking information.
# Survey for alignment researchers!

***UPDATE 3/9**: Thanks to broad participation from the community, this and associated surveys have raised approximately $10,000 for high-impact alignment organizations. Given the reasonable sample size we now have, we are now going to pause donations for any subsequent responses. (However, we will preserve the charity voting question, and if anyone wants to sponsor donations for any surveys taken after March 9th, please ping us at alignment@ae.studio.)*

[AE](http://www.ae.studio) [Studio](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment) is launching a [**short, anonymous survey**](https://docs.google.com/forms/d/e/1FAIpQLSfDzx7gqlzmvuHl4xKMZkoAGHEZeZMNulcTr_ykbHqmP1IC5g/viewform?usp=sf_link) for alignment researchers, in order to develop a stronger model of various [field-level](https://www.lesswrong.com/posts/zaaGsFBeDTpCsYHef/shallow-review-of-live-agendas-in-alignment-and-safety) dynamics in alignment. 

This appears to be an interestingly neglected research direction that we believe will yield specific and actionable insights related to the community’s technical views and more general characteristics. 

The survey is a straightforward 10-15 minute Google Form with some simple multiple choice questions.

For every alignment researcher who completes the survey, **we will donate $40 to a high-impact AI safety organization of your choosing **(see specific options on the survey). We will also send each alignment researcher who wants one **a customized report that compares their personal results to those of the field**. 

Together, we hope to not only raise some money for some great AI safety organizations, but also develop a better field-level model of the ideas and people that comprise alignment research. 

We will open-source all data and analyses when we publish the results. Thanks in advance for participating and for sharing this around with other alignment researchers! 

Survey full link: [https://forms.gle/d2fJhWfierRYvzam8](https://forms.gle/d2fJhWfierRYvzam8)
# London rationalish meetup: Feedbackloop-first rationality

> What?  
> This month's Rationalish will be hosted by Evgeny and Leo. We will be trying to implement some ideas from the blog post on Feedbackloop-first Rationality. We will alternate between trying 'Thinking Physics' problems and reflecting on the efficacy of the cognition used.  
>   
> ​Testing specific 'cognitive techniques' on the 'Thinking Physics' problems is encouraged, and information on some will be provided. Feel free to find / design your own to test too. (Some examples at the bottom).  
>   
> ​When?  
> This will go from 3pm - 5pm. But you can show up from 2pm, and stay later to chat.  
>   
> ​Who?  
> This event is for anyone interested in improving their rationality, no matter your current level. Having an understanding of some of the core ideas of (applied) rationality will help you get more out of the event.  
>   
> ​Reading list (Phil assumes this is optional):  
> Feedbackloop-first Rationality ([https://www.lesswrong.com/posts/pZrvkZzL2JnbRgEBC/feedbackloop-first-rationality](https://www.lesswrong.com/posts/pZrvkZzL2JnbRgEBC/feedbackloop-first-rationality))  
>   
> ​Some examples of 'cognitive techniques':
> 
> *   Noticing Confusion ([https://www.lesswrong.com/s/zpCiuR4T343j9WkcK/p/5JDkW4MYXit2CquLs](https://www.lesswrong.com/s/zpCiuR4T343j9WkcK/p/5JDkW4MYXit2CquLs))
> *   Resolve Cycles ([https://www.lesswrong.com/posts/jiJquD34sa9Lyo5wc/resolve-cycles](https://www.lesswrong.com/posts/jiJquD34sa9Lyo5wc/resolve-cycles))
> *   Tuning your Cognitive Strategies ([https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/tuning-your-cognitive-strategies](https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/tuning-your-cognitive-strategies))
> *   Internal Double Crux ([https://www.lesswrong.com/s/KAv8z6oJCTxjR8vdR/p/x2KrcscqgKDk6pMeD](https://www.lesswrong.com/s/KAv8z6oJCTxjR8vdR/p/x2KrcscqgKDk6pMeD))
> *   Murphyjitsu ([https://www.lesswrong.com/s/d922gAtBM8JNzkKKJ/p/zZkMm6ySRJKhHJHfq](https://www.lesswrong.com/s/d922gAtBM8JNzkKKJ/p/zZkMm6ySRJKhHJHfq))
# How has internalising a post-AGI world affected your current choices?

You're probably in one of the three groups below.

Group 1: You don't think AGI is coming anytime soon, so it is business as usual for you.

Group 2: You do think it is coming soon, but you haven't made any life changes beyond career choice and donations. I'm not that interested in hearing about career choices and donations (examples below for what I'd prefer to hear about).

Group 3: You do think it is coming soon, and you have made noteworthy life changes. 

I would *mostly* like to hear from Group 3. 

A quick brain dump of the kinds of things I am pointing at:

1.  You aren't saving for retirement. Heck, you aren't saving for 2030.
2.  You've decided not to buy property and rent instead. Or you have, with mortgage conditions that you'd normally wouldn't have.
3.  You've changed stock investment strategies.
4.  You're going to have kids / you're not going to have kids.
5.  You smoke cigarettes now, or don't wear sunscreen. You don't worry about cardiovascular health, or the long-term impact of drugs (I'm kind of tempted to take supplements for weight lifting I normally wouldn't TBH).
6.  You're spending much more time with family and friends, because they might be gone in X years.
7.  You've become more spiritual on purpose (maybe as a coping mechanism).
8.  You don't invest in learning about X topic or building skill Y, because they take ages to pay off.

Thanks!  
Yanni
# Safe Stasis Fallacy

A critical failure mode in many discussions of technological risk is the assumption that maintaining status quo for technology would lead to maintaining the status quo for society. Lewis Anslow [suggests](https://twitter.com/LouisAnslow/status/1693014091638415745) that this "propensity to treat technological stagnation as safer than technological acceleration" is a fallacy. I agree that it is an important failure of reasoning among some EAs, and want to call it out clearly.

One [obvious example](https://twitter.com/LouisAnslow/status/1691515081081704448) of this, flagged by Anslow, is the anti-nuclear movement. It was not an explicitly pro-coal position, but because there was continued pressure for economic growth, the result of delaying nuclear technology wasn't less power usage, it was more coal. To the extent that they succeeded narrowly, they damaged the environment.  

The risk from artificial intelligence systems today is very arguably significant, but stopping future progress won't reduce the impact of extant innovations. Stopping where we are today would still lead to continued problems with mass disinformation assisted by generative AI, and we'll see continued progress towards automation of huge parts of modern work even without more capable systems as the systems which exist are deployed in new ways. It also seems vanishingly unlikely that the pressures on middle class jobs, artists, and writers will decrease even if we rolled back the last 5 years of progress in AI - but we wouldn't have the accompanying productivity gains which could be used to pay for UBI or other programs.  

All of that said, the perils of stasis aren't necessarily greater than those of progress.  This is not a question of safety versus risk, it is a risk-risk tradeoff. And the balance of the tradeoff is debatable - it is entirely possible to agree that there are significant risks to technological stasis, and even that AI would solve problems, and still debate what strategy to promote - whether it is safer to [accelerate through the time of perils](https://forum.effectivealtruism.org/posts/i5cuLZH3SQJigiHMs/charting-the-precipice-the-time-of-perils-and-prioritizing-x), promote [differential technological development](https://michaelnotebook.com/dtd/index.html), or to [shut it all down](https://forum.effectivealtruism.org/posts/nWJLSjZ5T6HW8CMER/pausing-ai-developments-isn-t-enough-we-need-to-shut-it-all).