Harley-ml/lesswrong huggingface dataset:
# Cooking Air Quality

Our stove doesn't have a hood that vents to the outside, so when we're cooking something smoky we put a box fan in the window blowing out. I happened to have left my [air quality meter](https://temtopus.com/collections/temtop-co2-monitor/products/temtop-m2000-2nd-generation-air-quality-monitor-for-pm2-5-pm10-particles-co2-hcho-temperature-humidity-settable-audio-alarm-data-export-recording-curve-easy-calibration) running, and got a neat picture of how well this is working when cooking something very smoky:

[![](https://www.jefftk.com/particle-levels-during-cooking.png)](https://www.jefftk.com/particle-levels-during-cooking-big.png)

Here's the floorplan, marking the stove, fan, open windows, and meter:

[![](https://www.jefftk.com/unit2-downstairs-with-fan-and-meter.png)](https://www.jefftk.com/unit2-downstairs-with-fan-and-meter-big.png)

When we started cooking the windows were closed. When we noticed the smoke we opened three of the windows and put the fan in one of them blowing out. You can see this on the chart, because CO2 (the yellow line) starts to fall. Around 6:20 we stopped cooking, and you can see it almost immediately in the pm2.5 level (blue line). The pm10 level (red line) is pegged at the maximum (1mg/m³) so it's not clear if this started falling at the same time, but they usually move together. You can also see around 7:15 when we stopped eating dinner (nine people, including two kids, one toddler, and one baby) and went outside, because CO2 levels fall again.

We can use that last phase to try to estimate how much the fan was doing. Zooming in on just that section and only showing CO2, here's actual CO2 vs modeling it as decreasing toward atmospheric by 18.5% per minute:

[![](https://www.jefftk.com/co2-decrease-empty-room.png)](https://www.jefftk.com/co2-decrease-empty-room-big.png)

I think this is equal to 11 ACH: changing out 18.5% of the air each minute means the amount of air changed each hour is 60 * 18.5%. If we guess the effective volume is 24x24x9 then that's consistent with the fan moving about 1,000 CFM which is plausible for a box fan.

While putting the box fan in the window does get the smoke out, pm2.5 levels still measured above an unhealthy level of 50µg/m3 for ~45min. This is a pessimistic estimate, since one of the dining room windows was open and the people were in the dining room, and the meter was in the living room without any open windows. Still, not so good. Ideally we'd have a proper exhaust hood: if the fan were pulling directly from where the smoke was being produced it probably would have taken more like 500 CFM to keep up with it, instead of it taking a fan twice as powerful to play catch-up. We don't have any plans to redo the kitchen right now, but it does need to happen at some point and this is something I'll keep in mind!

In the mean time, aside from trying to remember to open the windows sooner, I'm going to try swapping out the fan for a more powerful one I have sitting spare in the basement. And probably also run the air purifier when cooking smoky things.

(We only cook things this smoky about once a month, so this is a relatively extreme example.)
# Replacing the Big Air Purifier

Around Christmas 2021 I put together a big air purifier on top of [our bookcase](https://www.jefftk.com/p/built-in-shelves). Somehow I didn't blog about it or take a picture, but here's it in the background of a birthday:

[![](https://www.jefftk.com/birthday-with-big-air-purifier.jpg)](https://www.jefftk.com/birthday-with-big-air-purifier-big.jpg)

The basic idea was that I didn't have a good place to put a filter cube, commercial air purifiers were scarce, and a different configuration of filters seemed like it could work well. It's a box fan with eleven filters: five along the side, five along the top, and one at the end. It glows because I built a regular light bulb into it, though without any means of replacing the bulb when it died (which it eventually did).

The design relies on a good seal with the wall, but when I initially set it up I was worried about peeling the paint off so I used masking tape there. Which eventually pulled away:

[![](https://www.jefftk.com/air-purifier-pulling-away-from-wall.jpg)](https://www.jefftk.com/air-purifier-pulling-away-from-wall-big.jpg)

Which means it's probably been doing about nothing for a while. I considered fixing it, but there were other issues with the design: the box fan was pretty loud on "high" and the fan didn't point a sensible direction. And at this point commercial purifiers are much more widely available. So I ordered three [AP-1518R](https://www.amazon.com/dp/B08LSC9HLS) purifiers, which are very similar to (and filter-compatible with!) the [AP-1512HH](https://www.amazon.com/dp/B01728NLRG) we already had, and replaced the big purifier:

[![](https://www.jefftk.com/new-air-purifiers.jpg)](https://www.jefftk.com/new-air-purifiers-big.jpg)

I've laid the purifiers on their backs, so the fresh air circulates towards people's heads and I can reach the controls from the ground. I'm not sure whether this is a good idea—the internal squirrel cage fan would have been engineered expecting loads from a different direction, and this might not be good for it? Any guesses?

I [previously tested](https://www.jefftk.com/p/evaluating-a-corsi-rosenthal-filter-cube) one of these against a filter cube and estimated a CADR of 219 CFM on high, 90% of what that filter cube gave on high. When the big purifier was new and sealed properly it was probably in this range or a bit worse: having nearly 3x the filters as a standard filter cube should have given less restriction on the airflow, but there's probably highly diminishing returns and the position of the outflow was bad. If I run the three new purifiers on medium they're far quieter than the big purifier while giving more clean air than it did before it started leaking. And I have the option to run them on high if the room is very crowded, where they're about the same volume as the big purifier while delivering about 3x the clean air.
# Apollo Neuro Follow Up

Two months ago I recommended the [Apollo Neuro](https://acesounderglass.com/2023/05/08/product-endorsement-apollo-neuro/). If you bought it, I would love it if you filled out [this extremely short form](https://docs.google.com/forms/d/1Kc3Gf1mj9KGqhIBqcCMpVjQYWath601pWXcvd0jEJps/edit) letting me know if it was useful to you. Results will go into an upcoming post to help other people decide if they should try it.
# Please speak unpredictably

Your predictable speech displeases both of us. I finish your sentences and am bothered by inefficiency. Thus, speak/write efficiently, to the point that I can't predict it.

Others may also dislike it as I do, but infer the rest at different levels.

Large audiences lower inference standards.

(I figure LessWrong can handle the compressed version)

This contradicts common advice. There is an optimum; do not go arbitrarily far. [Perhaps reverse this.](https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/)
# Which rationality posts are begging for further practical development?

Which posts seem to you like they have really important practical implications for how we ought to think and act, but don't seem to come with enough information about how to use the skills, or to develop them, or even what exactly the relevant skills are?  
  
Context: I'd like to devote many of my hours in the upcoming few months to [naturalist](https://www.lesswrong.com/tag/naturalism) study of topics that are critical to fully understanding the most important existing resources in rationality. I mainly have my eye on posts from Eliezer's Sequences, but I'm also open to other essays, books, and even videos. I'll be taking extensive notes, and turning them into companion pieces to complement the original essays. (See "[Lies, Damn Lies, and Fabricated Options](https://www.lesswrong.com/posts/gNodQGNoPDjztasbh/lies-damn-lies-and-fabricated-options)" and "[Investigating Fabrication](https://www.lesswrong.com/posts/NjZAkfio5FsCioahb/investigating-fabrication)" for an example of what I'm proposing.)
# A reformulation of Finite Factored Sets

I present my bachelor's thesis that reformulates [Finite Factored Sets](https://arxiv.org/abs/2109.11513):

[Causality with Deterministic Relationships](https://drive.google.com/file/d/1NxDozNx5EwX49mJ_sKhUrmUgxuIuUZoo)

Abstract:  
*We present a reformulation of Finite Factored Sets that uses functions and cartesian products instead of partitions and factorizations. In the new setting, we motivate assuming conditional orthogonality from observed conditional independence and show that inference is decidable. We show that Pearlian causal structures have analogs in the developed framework and propose an extension to the infinite unconditional case using measurable spaces.*

$$
\newcommand{\bigtimes}{\mathop{\Large\times\normalsize}}
$$

I will quote Section 1.3 to give an overview of the additions to the original paper.

Most of the concepts in this work were first introduced in \[[Finite Factored Sets](https://arxiv.org/abs/2109.11513) (FFS)\], where the author starts with a set $S$ and factorizes it using a set of partitions $\{F_i: i \in I\}$ of $S$. Then partitions on $S$ are analyzed using this factorization. This is called a *(finite) factored set*.

In this work, we instead start from the factors $F_i$ as sets and construct $F=\bigtimes_{i \in I} F_i$ as a cartesian product. Here we call $F$ a *(finite) factor space*. Instead of a partition on $S$, we analyze features $X: F \to \Omega_X$, where a feature would correspond to the partition $\{X^{-1}(\omega):\omega \in \Omega_X\}$ of $F$.

In the same manner, we can translate all concepts using functions. In the authors' opinion, this is a more natural way to frame things. We hope that this reframing is helpful to understand the framework and for doing further work.

To compare this work to the original paper \[FFS\], we give a detailed list of the main additions, apart from the reformulation.

*   In Section 2.4, we provide an alternative way to define conditional histories using a factor space induced on a subset $A \subseteq F$.  
    This is helpful to understand the concepts more deeply and leads, in some cases,  
    to shorter and more intuitive proofs.
*   In Section 3.3, we generalize the examples provided in \[FFS\] (Example 1 and 2)  
    to groups.
*   In Section 3.2 and the appendix, we spell out the implications carried by  
    the fundamental theorem of finite factored sets \[FFS\] (Theorem 3)  
    to justify assuming conditional orthogonality from observed conditional independence.
*   In Section 3.4, we prove that inference is decidable, thus solving  
    this open problem raised in \[FFS\] (Section 7.1).
*   In Section 4, we convert Pearlian causal structures and models to factor spaces.  
    We prove that d-separation in a causal structure is equivalent to  
    conditional orthogonality in the corresponding factor space.  
    We further give a definition of interventions in such a factor space and prove that they lead to the same probabilities a causal structure provides.
*   In Section 5, we provide a generalization of the framework to arbitrary  
    measure spaces in the unconditional case.
# Open problems in activation engineering

[Steering GPT-2-XL by adding an activation vector](https://www.lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector) introduced 

> **activation engineering**... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.

These results were recently complemented by [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://www.lesswrong.com/posts/kuQfnotjkQA4Kkfou/inference-time-intervention-eliciting-truthful-answers-from), which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes! 

We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from [the list](https://coda.io/@alice-rigg/open-problems-in-activation-engineering): 

*   Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture something meaningful?
*   Take a circuit studied from existing literature on GPT2, or find another one using [ACDC](https://github.com/ArthurConmy/Automatic-Circuit-Discovery/tree/main). Targeting the nodes in these circuits, can you learn anything more about them and generally about how activation additions interact with circuits?
*   What's the mechanism by which adding a steering vector with too large a coefficient breaks the model? ([Credit: Thomas Kwa](https://www.lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector?commentId=sAzDPXoQmAxxoKCi2); see also [@Ulisse Mini](https://www.lesswrong.com/users/ulisse-mini?mention=user)'s [initial data/explanation](https://www.lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector?commentId=FEGguDMzGbKojSQF9).)

If you want to work on activation engineering, come by the Slack server to [coordinate research projects and propose new ideas](https://join.slack.com/t/activationengineering/shared_invite/zt-1z82cqbrp-2YGwmTpYzQz3jTkPUXeNHg).
# Secure Hand Holding

When Lily as about three we were waiting at a crosswalk and traffic stopped for us. As we started across the street a driver that had been waiting to turn left misinterpreted the situation and, thinking traffic had stopped for them instead, tried to turn through our location. I tightened my grip and ran, pulling her through the air. The car stopped, about five feet past where it would have collided with us. Everyone was fine, but we were very shaken. And I was glad we'd been holding hands securely.

When we're walking around at the park or another relaxed environment there are a range of ways I'll hold hands with little kids. A common one is that I'll extend a single finger, and they'll wrap their hand around it. But when we're crossing the street, climbing stairs, or I otherwise might suddenly need to hold them up entirely, I additionally wrap my fingers loosely around their wrist:

[![](https://www.jefftk.com/secure-hand-holding-jeff.jpg)](https://www.jefftk.com/secure-hand-holding-jeff-big.jpg)

Another angle:

[![](https://www.jefftk.com/secure-hand-holding-b-jeff.jpg)](https://www.jefftk.com/secure-hand-holding-b-jeff-big.jpg)

The specific hold isn't that important: what matters is that it's comfortable for you both and if you had to you could really hold on. Julia finds that with the smaller size of her hands a different grip works better:

[![](https://www.jefftk.com/secure-hand-holding-julia.jpg)](https://www.jefftk.com/secure-hand-holding-julia-big.jpg)

I suspect many of you are thinking "of course, how else would you do it?" Many parents end up doing this pretty naturally. But if you currently are choosing between a friendly but weak companionship grip and securely holding their wrist this could be a nice thing to try!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid05RwiasYZFddV8pCPfzV743TVv1KDgR1299o65h217KKSnoX2SasJsDfYY5fvD7ryl), [mastodon](https://mastodon.mit.edu/@jefftk/110772152837425015)*
# The First Room-Temperature Ambient-Pressure Superconductor

From the post: 

> For the first time in the world, we succeeded in synthesizing the room-temperature superconductor ( K, 127C) working at ambient pressure with a modified lead-apatite (LK-99) structure. The superconductivity of LK-99 is proved with the Critical temperature (), Zero-resistivity, Critical current (), Critical magnetic field (), and the Meissner effect. The superconductivity of LK-99 originates from minute structural distortion by a slight volume shrinkage (0.48 %), not by external factors such as temperature and pressure. The shrinkage is caused by Cu substitution of Pb(2) ions in the insulating network of Pb(2)-phosphate and it generates the stress. It concurrently transfers to Pb(1) of the cylindrical column resulting in distortion of the cylindrical column interface, which creates superconducting quantum wells (SQWs) in the interface. The heat capacity results indicated that the new model is suitable for explaining the superconductivity of LK-99. The unique structure of LK-99 that allows the minute distorted structure to be maintained in the interfaces is the most important factor that LK-99 maintains and exhibits superconductivity at room temperatures and ambient pressure.
# Pulling the Rope Sideways: Empirical Test Results

Robin Hanson wrote [this post](https://www.overcomingbias.com/p/policy_tugowarhtml) a while back, which has since developed into a sort of inspiring slogan for how to achieve policy change: "Pull the Rope Sideways"

> The policy world can thought of as consisting of a few Tug-O-War "ropes" set up in this high dimensional policy space.  If you want to find a comfortable place in this world, where the people around you are reassured that you are "one of them," you need to continually and clearly telegraph your loyalty by treating each policy issue as another opportunity to find more supporting arguments for your side of the key dimensions.  That is, pick a rope and pull on it.   
> 
> If, however, you actually want to improve policy, if you have a secure enough position to say what you like, and if you can find a relevant audience, then prefer to pull policy ropes sideways.  Few will bother to resist such pulls, and since few will have considered such moves, you have a much better chance of identifying a move that improves policy.  On the few main dimensions, not only will you find it very hard to move the rope much, but you should have little confidence that you actually have superior information about which way the rope should be pulled.

I recently wrote in a casual slack channel:

> Hanson introduced the term "pull the rope sideways" and I think it's a good concept. However the metaphor bugs me a bit. In a tug-o-war, is it actually the case that pulling sideways causes the rope to move much farther that if you pull the rope in one of the normie directions? Has anyone tested this? It's not obvious to me why this would be true.

Vigorous debate ensued. We bought some rope and did a quick test.

Contrary to the expectations of most people in the thread, pulling the rope sideways didn't seem to work -- the overall effect seemed to be about the same as pulling it in one of the standard directions.  
  
Afterwards we joked about the implications for EA macrostrategy. "I guess we should just... pick a side." :D
# The Parable of the Dagger - The Animation

This video is an animation of [The Parable of the Dagger](https://www.lesswrong.com/posts/hQxYBfu2LPc9Ydo6w/the-parable-of-the-dagger), by Eliezer Yudkowsky. It's the first article of [A Human's Guide To Words](https://www.lesswrong.com/s/SGB7Y5WERh4skwtnb). 

The narration is almost word for word. The only discrepancies between the narration and the original text are some deletions of words such as "replied" or "said", which are redundant in an animation. But one remains intact:

> "How?!" cried the jester in horror, as he was dragged away. "It's logically impossible!"

The rationale for leaving that unchanged is that I'd like to modify the original articles as little as possible, so I felt iffy about deleting so many words, even if the visuals convey a similar meaning. Moreover, having "cried the jester in horror" makes clear to the viewer that the video is adapted from something you can read somewhere, not an original story written for Rational Animations. That said, I'm not sure it was the right decision in the end, although it probably doesn't matter much.

There is one actual modification to the parable, although it doesn't change its meaning. It is a small non-narrated addition at the end. The video would be way too sad without it.

We've also included a brief call to action on screen: "Read this story, and many more at [readthesequences.com](https://readthesequences.com)", plus some more links and details in the description and pinned comment. [For now](https://www.lesswrong.com/posts/TNsLdAZmSzMHTeaKf/should-rational-animations-invite-viewers-to-read-content-on), I'll direct people to that site instead of directly here.
# If I showed the EQ-SQ theory's findings to be due to measurement bias, would anyone change their minds about it?

Background on EQ-SQ
-------------------

The EQ-SQ theory is a theory by Simon Baron-Cohen which states that there's a biological sex difference in tendency to empathize with people (Empathizing) versus try to understand deterministic systems (Systematizing), and that autism represents the extreme Systematizing end of this spectrum.

Measurement bias
----------------

Since writing [this subthread](https://www.lesswrong.com/posts/ZjXtjRQaD2b4PAser/a-hill-of-validity-in-defense-of-meaning?commentId=mhnZjEjj9bdHxSFkv) where I found ambiguous and partial evidence that the EQ-SQ theory's findings are due to measurement bias, I've been considering collecting direct evidence with SBC's EQ and SQ-R scales, and write up a detailed post investigating measurement bias.

Measurement bias, if it exists ([as it very much seems to do on e.g. the EQ scale](https://www.lesswrong.com/posts/ZjXtjRQaD2b4PAser/a-hill-of-validity-in-defense-of-meaning?commentId=EWnJ2yWS7cKdbj8vy)), would most likely involve observing that the sex difference on the scales is limited to one subset of the items, the autistic-allistic difference is limited to another subset of the items, and maybe also the main thing the scale measures exists on a yet third subset of the items.

That is, it would involve observing that the scales mix different things together, and that autism and sex have distinct relationships with the things that get mixed together. As an analogy, imagine showing that autism is extreme masculinity by creating a "maleness test" which mixes together physical strength and poor social skills. Yes, this would have a large sex difference and a large autistic-allistic difference, but that'd transparently be measurement bias; the reason for the large sex difference (it measures physical strength) would be distinct from the reason for the large autistic-allistic difference (it measures social skills).

My concern
----------

My concern is that people didn't start believing in the EQ-SQ theory based on statistical correlations found with Simon Baron-Cohen's scales. They presumably started believing in it based on fuzzy intuitions arrived at through social experience.

So I could imagine that if I did show SBC's findings to be due to measurement error, people would sort of dismiss it as poorly designed scales and still continue believing in the theory. Like basically maybe SBC's studies provide a [faux-objective](http://benjaminrosshoffman.com/authoritarian-empiricism/) criterion they can use to throw at feminists, but if the criterion doesn't validate sex differences then EQ-SQ enthusiasts would [discard it and pick a new one](https://www.lesswrong.com/posts/TGux5Fhcd7GmTfNGC/is-that-your-true-rejection).

Thoughts?
# "Building a House" Review

I was recently reading Byron Barton's 1981 book, [Building a House](https://www.amazon.com/Building-House-Mulberry-Books-Barton/dp/0688093566). While it claims to be an end-to-end overview of the process of modern (for the time) home construction, there are enough errors in the illustrations that I wouldn't recommend it as a basic text.

For example, here's how they show installing a subfloor:

[![](https://www.jefftk.com/building-a-house-floor.jpg)](https://www.jefftk.com/building-a-house-floor-big.jpg)

There are several issues with the depicted method. The biggest one is that the seams do not fall on joists. This leaves the ends unsupported. The diagram shows nails at the joints, but those nails are doing nothing: they go through the panels into empty space. If your joist spacing doesn't match your panels you need to trim them.

Or, here's how they show framing the exterior walls:

[![](https://www.jefftk.com/building-a-house-exterior-wall-framing.jpg)](https://www.jefftk.com/building-a-house-exterior-wall-framing-big.jpg)

That window is not framed correctly. Not only is the header a single horizontal 2x when it should be a pair of vertical ones, but they've left off the jack studs entirely.

They also show the electrician installing ungrounded ("two prong") outlets:

[![](https://www.jefftk.com/building-a-house-electrical.jpg)](https://www.jefftk.com/building-a-house-electrical-big.jpg)

Grounded outlets have been required since the [1975 NEC](https://forums.mikeholt.com/threads/old-code-1971-and-modern-code.43704/), and had been best practice in new construction for about a decade.

Later they show installing pre-hung windows, which is fine, except that they say to do this after the drywall has already been installed and the hardwood floors have been laid:

[![](https://www.jefftk.com/building-a-house-installing-windows.jpg)](https://www.jefftk.com/building-a-house-installing-windows-big.jpg)

There are a bunch of reasons why you'd want to install the windows earlier, but a big one is that until your windows are installed any interior finishings are exposed to the elements.

The book is also seriously incomplete, with no discussion of insulation, drywall, heating, siding, and other core issues.

While the book continues to be a popular introduction to the topic, with [20 copies](https://find.minlib.net/iii/encore/search/C__Sbuilding%20a%20house%20barton__Orightresult__U?lang=eng&suite=cobalt&fromMain=yes) available in my local library network, I can't recommend it as a reference.
# Watermarking considered overrated?

_Status: a slightly-edited copy-paste of [a ~~Twitter~~ X thread](https://twitter.com/dfrsrchtwts/status/1682529184881709056) I quickly dashed off a week or so ago._

Here's a thought I'm playing with that I'd like feedback on: I think [watermarking](https://arxiv.org/abs/2301.10226) large language models is probably overrated. Most of the time, I think what you want to know is "is this text endorsed by the person who purportedly authored it", which can be checked with digital signatures. Another big concern is that people are able to cheat on essays. This is sad. But what do we give up by having watermarking?

Well, as far as I can tell, if you give people access to model internals - certainly weights, certainly logprobs, but maybe even last-layer activations if they have enough - they can bypass the watermarking scheme. This is even sadder - it means you have to strictly limit the set of people who are able to do certain kinds of research that could be pretty useful for safety. In my mind, that makes it not worth the benefit.

What could I be missing here?
1. Maybe we can make watermarking compatible with releasing model info, e.g. by baking it into the weights?
2. Maybe the info I want to be available is inherently dangerous, by e.g. allowing people to fine-tune scary models?
3. Maybe I'm missing some important reasons we care about watermarking, that make the cost-benefit analysis look better? E.g. avoiding a situations where AIs become really good at manipulation, so good that you don't want to inadvertently read AI-generated text, but we don't notice until too late? 

Anyway there's a good shot I don't know what I'm missing, so let me know if you know what it is.

Postscript: Someone has pointed me to [this paper](https://arxiv.org/abs/2012.08726) that purports to bake a watermark into the weights. I can't figure out how it works (at least not at twitter-compatible speeds), but if it does, I think that would alleviate my concerns.
# Untangling Infrabayesianism: A redistillation [PDF link; ~12k words + lots of math]

\[Epistemic status: improved redistillation of the [infrabayesianism sequence](https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa).\]

So you want to understand infrabayesianism, to hack to the center of that thorny wood and seek out and recover any treasures hidden there? You've come to a correct creature for a guide. If you want to journey there, make sure you've already got the necessary tools well in hand: some simple decision theory, the basics of topology and linear algebra, and a little measure theory - for that last, if you know how a Lebesgue integral is defined and why no reasonable σ-algebra can encompass the full power set, then you're already doing fine. If you find yourself struggling with such things, reach out to me on Discord or in PMs here and I'll see what we can do.

Infrabayesianism seems like exactly what we might need as alignment researchers: a way to discuss all of our usual decision-theoretic questions while also getting to account for uncertainty about the world, compensate for policy-dependent environments and adversarial selection, and even talk about UDT puzzles. It does this by fundamentally being a decision theory that has explicit reasonable machinery for handling Knightian uncertainty about the environment due to nonrealizable or nonlearnable hypotheses while still permitting nontrivial inference and planning.

Three major brambly hedges block the way between you and understanding: the prickly snagging of the frequently unclear, unintuitive, or just plain lacking notation used in the original infrabayesian sequence; thorny philosophical tangles up front and scattered throughout; and math and its accompanying density of concept and notation getting thicker as we go deeper in. Follow me, though, and we'll slip right through them with barely a scratch, and eat a couple of delicious berries from right off their vines. In fact, I can tell you up front that if you haven't read the original infrabayesianism sequence too closely and aren't that familiar with its notation... that's an active benefit, because we won't need most of it here. We won't be cleaving perfectly to its choices of notation or terminology, though I will eventually provide a dictionary between the two as a postscript.

(I *really* don't feel like trying to port over 28 PDF pages' worth of TeX and dense math writing here, let alone figure out how to divide it up into several posts, so rather than be redundant I've decided to link to the pdf instead. Please imagine that it was reproduced here in its entirety, and comment accordingly. This is a first-draft redistillation, of the kind that I might have been satisfied with as a minimum viable submission to the [Infra-Bayes bounty](https://www.lesswrong.com/posts/ig62gqK2xRPv79Wsr/usd500-bounty-contest-explain-infra-bayes-in-the-language-of). I may well edit, reorganize, revise, or remove parts of it as I see fit. If I feel like it, I might even turn this into an ArXiv submission or sequence of carefully-ordered posts of my own.)
# “Desperate Honesty” by Agnes Callard

An excellent meditation on intellectual humility as an essential tool in the philosopher's toolbox, and how valuable it is to us when people poke holes in our favorite ideas. Excerpts:

> These days I regularly wander outside the gates of philosophy, and it is remarkable to me how often I receive applause for doing what comes absolutely as a matter of course within our discipline. Those praising me seem to imagine that I am constantly lashing myself with the painful whip of intellectual morality.

⋮

> The encounter with the mind of a person who is determined to let you get away with absolutely nothing, and who has been extensively trained to succeed at precisely that task—I can’t decide whether that is like stepping in front of a firing squad, or stepping into a cathedral.

⋮

> \[M\]y students are somehow already equipped with opinions, which, if I’m doing my job right, will in short order get launched at me in the form of questions, objections, and counterexamples. Philosophical questions, at least those that matter most to us, show up in the mind already answered. We recognize these questions, they spark of familiarity, because they lie at the foundation of what we’ve been doing all along, and how we have been thinking all along...
> 
> Given where my students start—which is not nowhere, but already somewhere—I can’t be in the business of telling them what to think. Rather, I’m forced to be in the business of inviting them to fight me. This is not true in other disciplines. A physics or Greek teacher can safely assume they are either starting with a clean slate, or working with a solid shared foundation. Either way, they’re on level ground, building upwards. The philosophy teacher looks out a landscape that is already full, and not entirely in a good way. In philosophy we don’t build up, at least not without tearing down. Our starting point is not the barren wastes of ignorance, but the clutter of falsity. Philosophy begins in error.

⋮

> Realizing you need \[your critics\] means realizing that they are the ones doing you a favor, that you had better make arguing with you not just palatable, but ideally so pleasant as for them to find it downright delightful. And so yes of course you are going to be a paragon of cheerfulness when they raise their objections, you will be “generous” with criticism, and “fair” to your critics, trying to extract every bit of help from their comments. They hold all the cards, and you hold none. You need them, they don’t need you.

⋮

> \[N\]orms that govern how people behave when they realize they truly need one another cannot be described as “intellectual” in any restricted sense. Philosophical honesty is a tool to that allows a person to lean on the minds of those around her; it makes a person socially adept.  Philosophers are the socializers par excellence, our skills are social skills.  And I want to use these skills all the time, in all my socializing. Not because that kind of behavior is more virtuous, or generous, or disciplined, but because what I see in the classroom, I see everywhere.
# What is ontology?

Over the years I've picked up on more and more phrases that people on LessWrong use. However, "ontology" is one of them that I can't seem to figure out. It seems super abstract and [doesn't seem to have](https://www.lesswrong.com/tag/ontology) a reference post.

So then, please [ELI5](https://www.urbandictionary.com/define.php?term=ELI5): what is ontology?
# "Is There Anything That's Worth More"

In season two, episode twenty-four of _Steven Universe_, ["It Could've Been Great"](https://steven-universe.fandom.com/wiki/It_Could've_Been_Great), our magical alien superheroine protagonists (and Steven) are taking a break from building a giant drill to extract a superweapon that was buried deep within the Earth by an occupying alien race thousands of years ago, which is predicted to emerge and destroy the planet soon.

While our heroines watch the sunset, Peridot (who alerted them to the buried superweapon) expresses frustration that the group isn't still working. Steven defends their leisure: "Working hard is important, but feeling good is important, too," he says. He then goads Peridot into [a musical number](https://www.youtube.com/watch?v=ss7rLjGAlQE), which includes a verse from her explaining her attitude towards the situation and her forced compatriots:

> _I guess we're already here  
> I guess we already know  
> We've all got something to fear  
> We've all got nowhere to go  
> I think you're all insane  
> But I guess I am, too  
> Anybody would be if they were stuck on Earth with you_

"It Could've Been Great" aired in 2016. At the time, I agreed with Peridot: with the fate of the planet on the line, our heroines and Steven should have been burning the midnight oil. If they succeeded at disarming the superweapon, they'd have plenty of time to rest up afterward, but if they failed, there would be no more time for them.

Now, as the long May 2020 turns into March 2023, I'm starting to think that Steven had a point.

It would be one thing if our heroines knew with certainty that the superweapon would go off at a given date and time, presenting a definite do-or-die deadline. But all they had to go on was Peridot's warning. Attempting a speculative technical project to avert uncertain doom with an uncertain deadline, their planning had to average over many possible worlds—including worlds where the problem of survival was too easy or too hard for their efforts to matter, such that even the utility of leisure in the present moment was enough to sway the calculation.
# Bad Imitation Instruments

It's common to look down on modern instruments as cheap imitations: the digital piano as an inferior piano, [foot drums](https://www.jefftk.com/p/introduction-to-heel-toe-drumming) as an inferior drum kit. While this isn't wrong, as far is it goes, it's a very limiting way to look at these instruments. Instead of writing them off for their faults, it's a lot better to figure out what they're good at and what new musical possibilities they open up.

Some historical perspective might be useful: there's a long line of classic instruments that were created as bad imitations of other instruments, that we now appreciate for themselves!

Consider the [Hammond](https://en.wikipedia.org/wiki/Hammond_organ) organ:

It was invented and originally marketed as a cheaper alternative to the pipe organ, and even lost an enforcement action from the FTC after advertising that it was an equivalent instrument. Which makes sense: if you evaluate it as a pipe organ, it's really not a very good one. And yet, appreciated for itself, it's an incredible versatile instrument, well worth playing in its own right.

Or the [Rhodes](https://en.wikipedia.org/wiki/Rhodes_piano) piano:

It's a similar story: what began as a cheap portable substitute for the piano in military hospitals is now a valued for its own complex sound.

Or the [DX7](https://en.wikipedia.org/wiki/Yamaha_DX7) keyboard:

Many of its presets are pretty lacking imitations of the instruments they're modeled after, but the DX7, played by people who wanted it for the sounds it did make, was tremendously influential in the mid-80s.

Others include the [Mellotron](https://en.wikipedia.org/wiki/Mellotron)/[Orchestron](https://en.wikipedia.org/wiki/Orchestron), [Harmonium](https://en.wikipedia.org/wiki/Pump_organ#Harmonium), and even the [electric guitar](https://en.wikipedia.org/wiki/Electric_guitar).

So I'd encourage you to keep an open mind when listening to new instruments, and focus on what they have to offer instead of whether they represent a strict improvement over more established options.
# [Linkpost] Deception Abilities Emerged in Large Language Models

*This is a linkpost for* [*https://arxiv.org/abs/2307.16513*](https://arxiv.org/abs/2307.16513)*. *

> Large language models (LLMs) are currently at the forefront of intertwining artificial intelligence (AI) systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, such as GPT-4, but were non-existent in earlier LLMs. We conduct a series of experiments showing that state-of-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can alter their propensity to deceive. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.

This seems to me like a very good target for (mechanistic) interpretability (though would require access to GPT-4/ChatGPT activations/weights).
# [Linkpost] Multimodal Neurons in Pretrained Text-Only Transformers

*This is a linkpost for* [*https://arxiv.org/abs/2308.01544*](https://arxiv.org/abs/2308.01544)*. *

> Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model's residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning.
# Private notes on LW?

Lately I've been noticing what a powerup is to read things in google docs, where I can take whatever notes I want as in-line comments without worrying about looking dumb or confusing. In changes my relationship to confusing passages, where I feel much more affordance to think through what exactly is confusing about it.

As a general reading-habit, "copy it into google docs" is a pretty good habit. But I (and I think others on LW team although for slightly different reasons) have been thinking about building a feature directly into LW to facilitate it. 

One version of it might explicitly be "private notes" that are optimized as such. 

Another version of it might basically just take the side-comment button we already have and add a "private comments" option that lets you set the comment to "everyone", "only you", "you + author" (for giving the author feedback in a way that's more private than a comment but having more context included than a DM) \[edit: also, sharing the comment with arbitrary people is a fairly obvious feature here\]

Curious what people think about this and what options they'd expect themselves to use.

I'm maybe specifically wondering whether people expect a UI that's oriented around "arbitrary sharing" would feel good enough as a personal note-taking thing.
# What are the best published papers from outside the alignment community that are relevant to Agent Foundations?

I am after papers that you have stumbled across that may be relevant to the core of the understanding the existential AI alignment problem despite not having explicit links to AI alignment. 

Here are papers I have found that fit the above criteria to give you a better idea of what I'm after:

*   [Primas, Hans. *Emergence in Exact Natural Science*. 1998](https://philpapers.org/rec/PRIEIE)
*   [Shalizi, Cosma. *Causal Architecture, Complexity, and Self-Organization in Time Series and Cellular Automata.* 2001](http://bactra.org/thesis/single-spaced-thesis.pdf)
*   [Jeremy, England. *Dissipative Adaptation in Driven Self-Assembly.* 2015](http://www.englandlab.com/uploads/7/8/0/3/7803054/nnano.2015.250__1_.pdf)

While I've used the term "agent foundations" I expect the majority of useful papers will not use terms like agency, optimization etc.
# [Linkpost] Applicability of scaling laws to vision encoding models

*This is a linkpost for* [*https://arxiv.org/abs/2308.00678.*](https://arxiv.org/abs/2308.00678.)

> In this paper, we investigated how to build a high-performance vision encoding model to predict brain activity as part of our participation in the Algonauts Project 2023 Challenge. The challenge provided brain activity recorded by functional MRI (fMRI) while participants viewed images. Several vision models with parameter sizes ranging from 86M to 4.3B were used to build predictive models. To build highly accurate models, we focused our analysis on two main aspects: (1) How does the sample size of the fMRI training set change the prediction accuracy? (2) How does the prediction accuracy across the visual cortex vary with the parameter size of the vision models? The results show that as the sample size used during training increases, the prediction accuracy improves according to the scaling law. Similarly, we found that as the parameter size of the vision models increases, the prediction accuracy improves according to the scaling law. These results suggest that increasing the sample size of the fMRI training set and the parameter size of visual models may contribute to more accurate visual models of the brain and lead to a better understanding of visual neuroscience.
# Stomach Ulcers and Dental Cavities

*(This is a linkpost from my blog, De Novo)*

Recently I learned about [an effort to prevent dental cavities](https://manifund.org/projects/recreate-the-cavity-preventing-gmo-bacteria-bcs3-l1-from-precursor-) by using genetically modified bacteria to outcompete cavity-causing bacteria. This got me thinking: why has the idea of preventing cavities by targeting bacteria not been more developed already?

The current situation reminds me of the history of stomach ulcers. Before the 1980s, doctors recommended avoiding spicy foods and reducing stress to alleviate stomach ulcers. However, once Robin Warren and Barry Marshall proved ulcers were due to *H. pylori* infection,[^wp6v2sn31ur] treatment with antibiotics to eliminate the bacteria became the standard of treatment.

Today, dentists recommend avoiding sugary foods and brushing your teeth to prevent cavities. But we know cavities are caused by bacteria (in particular *Streptococcus mutans),* so why not directly attack cavity-causing bacteria?

Some potential ideas:

*   Selectively targeted antibiotics
*   Vaccines ([previously tried in the 1980s](https://link.springer.com/article/10.1007/BF00146392), not very successful because it’s difficult to get antibodies to penetrate biofilms, and also because *S. mutans* has several different strains with different antigenic profiles)
*   Outcompeting *S. mutans* with different bacteria (the current effort by Aaron Silverbook, which I think is promising)

Basically, what Aaron Silverbook [is proposing to do](https://manifund.org/projects/recreate-the-cavity-preventing-gmo-bacteria-bcs3-l1-from-precursor-) is recreate a strain of *S. mutans,* termed BSC3-L1, that is deficient in lactic acid production. This was previously developed by a company called Oragenics, but they abandoned the effort (I think due to financial reasons). It seems Aaron’s team is mostly people from software backgrounds, so they would probably appreciate help from any talented microbiologists who happen to be reading this post.

[^wp6v2sn31ur]: In a famous case of self-experimentation, Marshall drank a culture of H. pylori and subsequently developed gastritis. For this work, Warren and Marshall earned the 2005 Nobel in Physiology/Medicine.
# how 2 tell if ur input is out of distribution given only model weights

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/31f1c57d6edbd1cbc24f3be02b08aa0ee11321822313d13c.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/86a7c9ea0db67630c8eae21ef6d8b44badc844b7f16b1f7a.png)

(Hastily-written code to reproduce these findings is available [here](https://pastebin.com/iEgQPB8r). It also contains some extraneous logic.)
# [Linkpost] Will AI avoid exploitation?

As some of you may know, I'm editing a special issue of the journal *Philosophical Studies* on AI safety (along with [@Dan H](https://www.lesswrong.com/users/dan-h?mention=user) ). I thought I'd share the first paper from the issue, which deals with some issues in AI safety theory that have been frequently discussed on LessWrong.

Here's the abstract:

*A simple argument suggests that we can fruitfully model advanced AI systems using expected utility theory. According to this argument, an agent will need to act as if maximising expected utility if they’re to avoid exploitation. Insofar as we should expect advanced AI to avoid exploitation, it follows that we should expect advanced AI to act as if maximising expected utility. I spell out this argument more carefully and demonstrate that it fails, but show that the manner of its failure is instructive: in exploring the argument, we gain insight into how to model advanced AI systems.*

You can find the paper here: [https://link.springer.com/article/10.1007/s11098-023-02023-4](https://link.springer.com/article/10.1007/s11098-023-02023-4)