Harley-ml/lesswrong huggingface dataset:
# Deep and obvious points in the gap between your thoughts and your pictures of thought

Some [ideas](https://worldspiritsockpuppet.substack.com/p/bodega-bay-workshop-18-11-26) feel either deep or extremely obvious. You’ve heard some trite truism your whole life, then one day an epiphany lands and you try to save it with words, and you realize the description is that truism. And then you go out and try to tell others what you saw, and you can’t reach past their bored nodding. Or even you yourself, looking back, wonder why you wrote such tired drivel with such excitement.

When this happens, I wonder if it’s because the thing is true in your model of how to think, but not in how you actually think.

For instance, “when you think about the future, the thing you are dealing with is your own imaginary image of the future, not the future itself”.

On the one hand: of course. You think I’m five and don’t know broadly how thinking works? You think I was mistakenly modeling my mind as doing time-traveling and also enclosing the entire universe within itself? No I wasn’t, and I don’t need your insight.

But on the other hand one does habitually think of the hazy region one conjures connected to the present as ‘the future’ not as ‘my image of the future’, so when this advice is applied to one’s thinking—when the future one has relied on and cowered before is seen to evaporate in a puff of realizing you were overly drawn into a fiction—it can feel like a revelation, because it really is news to how you think, just not how you think a rational agent thinks.
# In set theory, everything is a set

In Unix, everything is a file; in set theory, everything is a set.

* * *

This abstraction is very cool and allows us to prove facts that would not be provable if we thought that sets could include stuff that were not sets and therefore could not be broken down further.

Here's a nice example:

I was asked to prove both of the following statements in *Naive Set Theory*

$$
 X=\bigcup \mathcal{P}(X) ; 
$$

$$
 X \subset \mathcal{P}\left (\bigcup X \right ). 
$$

When I try to prove something, especially in set theory, I try to visualize it in my head to make an intuitive proof and then write the mathematical proof after. I was able to easily visualize the first statement. Since the power set is just all of the subsets of a set, surely unioning all of the subsets of a set would give the set itself.

The second one was much more confusing and I couldn't visualize it. It’s basically saying, take a set, union all its elements together, then take the power set of *that*, and the original set will be a subset of that power set. I got lost at union all of its elements together. How does this work? For example, how do we do $\bigcup \{ 1, 5, 113, 55 \}$ ? The elements are just numbers, not sets! … Oh wait, the beginning of the book said that in set theory **all we have are sets**. So they are actually sets!

For each item in $X$, there are two possibilities of what it could be.

It could either be the empty set ($\varnothing$), or a set with items in it (literally everything else including numbers, ordered pairs, functions, etc). If it is the empty set, no elements will be added to the union, but the empty set is always an element of any power set, because it is a subset of all sets. So that element of $X$ is taken care of.

For the rest of the elements of $X$ that were *not* the empty set, their underlying elements will get merged together in $\bigcup X$. For example, let’s say $X = \{ \varnothing, \{a,b \}, \{ c\} \}$. Then $\bigcup X = \{a,b,c\}$. The key thing is that taking the power set of this will create **all the sets that were originally in **$X$**, and more. **$\mathcal{P} \left ( \bigcup X \right ) = \{\varnothing, \{a\}, \{b\}, \{c\}, \{a,b\}, \{a,c\},\{b,c\},  \{a,b,c\}\}$**.** This contains all of the sets that were originally in $X$ plus a few more. Once we have the everything is a set mindset, then it becomes visually intuitive why $X \subset \mathcal{P} \left ( \bigcup X\right )$: we are re-building all of the sets that made up $X$, and more. A simple change of perspective turned an unintuitive problem into an intuitive one! I love this.

* * *

**Why am I writing about this?**

When I find a cool idea that challenges my previous thinking, I like to blog about it so that I can think even more clearly about it. I also like to do it because I think more people should know that everything is a set!
# Balancing Games

When I play an N-player game I want everyone to both:

*   Try to win
*   Win about 1/N of the time

With many games and groups of participants these are in conflict: if I play bridge against my kids I'm going to win all the time, but I'm not very good at the game so if I play against people who are serious about it I'm going to lose ~all the time.

One way some games handle this is by including a lot of luck. The more random the outcomes are, the more you'll approach 1/N regardless of player skill. Kid games where you make no choices, like [Candyland](https://www.jefftk.com/p/candyland-is-terrible) or War, take this to the extreme.

Instead, I think [handicapping](https://en.wikipedia.org/wiki/Handicapping) is a much better approach. For example in Go the weaker player can [start with several stones already on the board](https://en.wikipedia.org/wiki/Handicapping_in_Go), which gives them an advantage while still keeping it interesting and without turning it into a different-feeling game. When I was little and playing Go with my dad I remember slowly reducing the number of handicaps I needed over months, which was really rewarding: each game was fun and challenging, and I could see my progress.

Other examples:

*   In Dominion, changing the ratio of coppers to estates that each player starts with.
    
*   In Settlers of Catan, allowing weaker players to place both of their settlements before stronger ones.
    
*   In Power Grid, Monopoly, Modern Art, or anything else financial, letting weaker players start with more money.
    
*   In Ticket to Ride, Thurn und Taxis, Settlers of Catan, or anything else with resource cards, letting weaker players start with more cards.
    

I like it when games are designed in a way that makes this kind of adjustment easy and granular. You can calibrate by removing a handicap after the weaker player wins some number of games in a row (I think three is about right though it depends on granularity) and vice versa.

I'm curious, though: why isn't this more common? It's very normal in Go, mostly of historical interest [in chess](https://en.wikipedia.org/wiki/Handicap_(chess)), and in most game cultures I'm around it seems like the expectation is just that weaker players will just lose a lot or or stronger players will "go easy" on them? Is it that acknowleging that some players are stronger than others is awkward? Too hard to calculate for games with more than two players?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0EtjeViaSLqd23DGe7pq9ZJHhDsdRpJ5BoQGFUc9mh2bmdcUn9dg5NuULX79286U6l), [mastodon](https://mastodon.mit.edu/@jefftk/111986951579892041)*
# Cellular respiration as a steam engine

When I helped an interlocutor learn about metabolism, we made an analogy between steam engines and cellular respiration. I find that people (at first) know steam engines better than cellular respiration, so I show the analogy here as a guide to cells assuming you understand engines. You could also use it in the other direction, if you learned about cells first.

If you go far enough into the details, the analogy will break down, as would any analogy. But this one holds more precisely than I or my interlocutor expected.

Glucose in the cell is like coal in the boiler. The boiler burns coal; likewise, the cell runs glucose thru the citric acid cycle (after glycolysis). The fire (the citric acid cycle and electron transport chain) heats the water (adenosine diphosphate, ADP) to a closely related, higher-energy form (adenosine triphosphate, ATP), which is steam. The steam (ATP) pushes a turbine (enables cellular work), condensing back into water (ADP).

With some changes to the engine, you can use oil in place of coal. Likewise, with some changes to what the cell does (beta-oxidation in place of glycolysis), it can use fat (or ketone bodies) in place of glucose. The fire of the boiler is essentially the same, and once the cell gets acetyl-CoA from carbs or fats, it uses the same citric acid cycle and ETC in either case.

Most of the boiler corresponds to the mitochondrion. The pipes carrying water to be boiled are like ATP synthase. ATP synthase is an enzyme embedded in mitochondrial membranes, which lets protons across the membrane in a way that phosphorylates ADP to ATP.

In some animals (including humans), the mitochondria in brown adipose tissue let pumped protons cross back for heat rather than ATP. This is like exposing the building around the steam engine to the heat of the boiler.
# Band Lessons?

Music lessons for individuals are common and normal, but what about for bands? It sounds a bit unusual, but it's something I've gotten a lot out of:

*   In early 2013, when the [Free Raisins](https://www.freeraisins.com/) had been playing for 2.5y and had played ~60 dances we had a session with Max Newman, then of [Nor'easter](https://noreasterdanceband.bandcamp.com/album/waiting-for-the-snow), now of the [Stringrays](https://www.stringraysmusic.com/). He was going to be coaching another band and wanted to practice on friends first. We talked about what made good dance music, how to support contra in particular, and about being intentional about the story arc of each dance and of the evening as a whole.
    
*   That spring we had another coaching session, this time with Lisa Greenleaf. Now, Lisa is a caller, and not a musician, but she has a great sense of what she wants from the music. I remember her taking turns pointing at a random one of us and saying "do something interesting!" so that we wouldn't sound the same each time through.
    
*   A year later, in Spring 2014, Andrew VanNorstrand, then of [Great Bear](https://en.wikipedia.org/wiki/Great_Bear_(band)), now of the [Faux Paws](https://thefauxpawsmusic.com/), produced our [CD](https://www.freeraisins.com/out-of-the-box). While this was primarily about putting together a good album and not improving as a band, we learned a ton and did get a lot better.
    
*   In Fall 2019, when [Kingfisher](https://www.kingfisherband.com/) had been playing for ~1y and had played ~20 dances, we had a session with Andrew. Even though I'd worked with him a lot a few years earlier on the CD, that was with a different band. What Kingfisher needed was pretty different: my memory is we were moving through ideas too quickly, we weren't listening to each other enough, we were thinking of rhythm as just me and melody as just Cecilia, and we were a bit all over the place.
    

I'm really glad we did these sessions and, while playing for a lot of dances is the best way to get good at playing for dances, a bit of time working with someone who has a bunch of experience, good judgement, good communication skills, and an outside perspective, can combine well with this.

If you're looking for a coaching session, in addition to Max, Lisa, and Andrew, other people that come to mind are [Audrey Jaber](https://www.audreyknuth.com/), [Amy Englesberg](https://www.amyenglesberg.com/), or [Julie Vallimont](https://julievallimont.com/). Note that I haven't talked with any of them about this post, and don't know anything about their current interest in coaching! I just know them and think this is something they're good at. You could also consider reaching out to most people who are [playing a lot of big events](https://www.jefftk.com/news/gigstats).
# Post series on "Liability Law for reducing Existential Risk from AI"

Gabriel Weil (Assistant Professor of Law, Touro University Law Center) wrote this post series on the role of **Liability Law for reducing Existential Risk from AI**.  I think this may well be of interest to some people here, so wanted for a linkpost to exist. 

The first post argues that [Tort Law Can Play an Important Role in Mitigating AI Risk](https://forum.effectivealtruism.org/posts/epKBmiyLpZWWFEYDb/tort-law-can-play-an-important-role-in-mitigating-ai-risk). 

The second post addressed directly [How Technical AI Safety Researchers Can Help Implement Punitive Damages to Mitigate Catastrophic AI Risk](https://forum.effectivealtruism.org/posts/yWKaBdBygecE42hFZ/how-technical-ai-safety-researchers-can-help-implement)

Here is the [full paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4694006).

TLDR (from the first post)

> Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that does not require provable negligence) and expanding the availability and flexibility of punitive damages is feasible, but will require action by courts or legislatures. Legislatures should also consider acting in advance to create a clear ex ante expectation of liability and imposing liability insurance requirements for the training and deployment of advanced AI systems. The following post is a summary of a law review article. [Here is the full draft paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4694006). Dylan Matthews also did an [excellent write-up of the core proposal](https://www.vox.com/future-perfect/2024/2/7/24062374/ai-openai-anthropic-deepmind-legal-liability-gabriel-weil) for Vox’s Future Perfect vertical.
# Supposing the 1bit LLM paper pans out

[https://arxiv.org/abs/2402.17764 claims that 1 bit LLMs are possible.](https://arxiv.org/abs/2402.17764)

If this scales, I'd imagine there is a ton of speedup to unlock since our hardware has been optimized for 1 bit operations for decades. What does this imply for companies like nvidia and the future of LLM inference/training? 

 Do we get another leap in LLM capabilities? Do CPUs become more useful? And can this somehow be applied to make training more efficient?

Or is this paper not even worth considering for some obvious reason I can't tell.   
 

Edit: this method is applied to training already
# Cryonics p(success) estimates are only weakly associated with interest in pursuing cryonics in the LW 2023 Survey

[The Less Wrong 2023 survey results are out](https://www.lesswrong.com/posts/WRaq4SzxhunLoFKCs/2023-survey-results). As usual, it includes some questions about cryonics. One is about what people’s level of interest in cryonics is (not interested, considering, [cryocrastinating](https://cryonics.miraheze.org/wiki/Cryocrastination), signed up, etc.). Another asks about people’s subjective probability of successful restoration to life in the future, conditional on there not being a global catastrophe destroying civilization before then. This is also known as p(success). I thought it might be interesting to plot these (with the subjective probability estimates on a log scale, of course):

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea32e658-9222-4e6d-b042-5183b3935066_1510x1476.png)

[R code available here](https://github.com/andymckenzie/R-plots/blob/master/lww_survey_2.r)

It is true that people who are more interested tend to give higher subjective probability estimates of success (median probability estimates: signed up = 17.5%, cryocrastinating = 30%, considering = 10%, not interested = 5%). But the difference is not very large. There must be other factors that are much more important than p(success) estimates in mediating whether someone is interested in signing up for cryonics and/or actually goes through with it.

This is crossposted from my blog's links post for the month, [available here](https://neurobiology.substack.com/p/action-potentials-for-february-995). I only posted this part because I thought it was less likely that people would be interested in the others.
# Elon files grave charges against OpenAI

> (CN) — Elon Musk says in a Thursday lawsuit that Sam Altman and OpenAI have betrayed an agreement from the artificial intelligence research company's founding to develop the technology for the benefit of humanity rather than profit.
> 
> In the [suit](https://www.courthousenews.com/wp-content/uploads/2024/02/musk-v-altman-openai-complaint-sf.pdf) filed Thursday night in San Francisco Superior Court, Musk claims OpenAI's recent relationship with tech giant Microsoft has compromised the company's original dedication to public, open-source artificial general intelligence.
> 
> "OpenAI, Inc. has been transformed into a closed-source de facto subsidiary of the largest technology company in the world: Microsoft. Under its new board, it is not just developing but is actually refining an AGI to maximize profits for Microsoft, rather than for the benefit of humanity," Musk says in the suit.
# Good HPMoR scenes / passages?

I'm doing a reading of good fan-fiction at a con this weekend, to counter the many "bad fanfic reading" panels.  I want to read an interesting passage from HPMoR, but I can't remember any particular passage myself, and I don't want to re-read the whole thing this week.  Can anyone remember any scene or passage that stuck in their mind as impressive?

It should:

*   be less than 1000 words
*   be understandable to someone who's read Harry Potter, but not HPMoR, with at most one or two sentences of background info (please write those sentences in your reply)
*   be good at the things that make HPMoR special, e.g.,
    *   a clever "exploit" of Muggle society using magic
    *   gobsmacking you with a new way of looking at something you thought you knew well
    *   convincing you that something that seems evil is the right thing to do
    *   Quirrell or Harry creeping you out, scaring you, or making you question your ethics by being ruthlessly rational
# Boston's Line 1

Over on r/mbta people [were speculating](https://www.reddit.com/r/mbta/comments/1b6co3w/if_the_mbta_switched_to_numbered_lines_what_would/): if Boston's rapid transit lines were numbered, like [some](https://en.wikipedia.org/wiki/Paris_M%C3%A9tro_Line_1) [other](https://en.wikipedia.org/wiki/Line_1_(Beijing_Subway)) [cities](https://en.wikipedia.org/wiki/U1_(Berlin_U-Bahn)), which line would get the honor of being first? The [Orange Line](https://en.wikipedia.org/wiki/Orange_Line_(MBTA)), as the main line of the historical [Boston Elevated Railway](https://en.wikipedia.org/wiki/Boston_Elevated_Railway)? The [Red Line](https://en.wikipedia.org/wiki/Red_Line_(MBTA)) , with the highest (pre-covid) ridership? The [Green Line](https://en.wikipedia.org/wiki/Green_Line_(MBTA)), as the oldest line (and the oldest subway in North America)? The [Blue Line](https://en.wikipedia.org/wiki/Blue_Line_(MBTA)), as, uh, the only line to run cars that switch between catenary and third rail?

It turns out there's an actual answer! From [1964](https://upload.wikimedia.org/wikipedia/commons/d/d3/1964_M.T.A._Boston_map.png) to [1966](https://www.mbtagifts.com/products/1966-mbta-system-route-map-fine-art-print), before switching to [colors](https://www.mbtagifts.com/products/1967-mbta-system-map-fine-art-print) in 1967, the MTA (and then the MBTA) used numbers. Here's the 1966 system map:

[![](https://www.jefftk.com/mbta-system-map-1966.jpg)](https://www.jefftk.com/mbta-system-map-1966-big.jpg)

The lines were:

1.  Harvard to Ashmont (Red Line, Ashmont Branch)
2.  Everett to Forest Hills (Orange Line)
3.  Bowdoin to Wonderland (Blue Line)
4.  Lechmere to Riverside (Green Line, D)

(I originally noticed the claim that they used to be numbered on Wikipedia, but then when I went looking for maps I couldn't find any examples and marked it as [citation needed](https://en.wikipedia.org/w/index.php?title=Red_Line_(MBTA)&diff=prev&oldid=1211820899). Then I found that the parallel claim on the Blue Line cited the (incredible!) [Changes to Transit Service in the MBTA district](http://roster.transithistory.org/MBTARouteHistory.pdf), which has this on p8. I realized that I'd been looking at maps that were too early, and when I checked the years immediately preceding the introduction of the colors I found it. I've now [updated](https://en.wikipedia.org/w/index.php?title=Red_Line_%28MBTA%29&diff=1211822373&oldid=1211820899) the page with a citation to the 1966 map.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02w3KRqqTBX9JX5eiDqgTkSWu6f12W4oKhs3JwooeZxEzfbhBgcr3pw9meL5MBLi5Kl), [mastodon](https://mastodon.mit.edu/@jefftk/112039070047446040)*
# Why correlation, though?

This is a very basic question to ask, but I'm not sure I actually understand some fundamental properties people seem to ascribe to correlation.

As far as I understand it, correlation usually refers to Pearson's correlation coefficient, which is, according to Wikipedia, "*a correlation coefficient that measures linear correlation between two sets of data*." Cool.

But then I see a discussion on whether variables X and Y are related in some way, and reads like:

*   well, X correlates to Y with r=0.8 so it's a good predictor/proxy/whatever
*   X and Y have been found to correlate with r = 0.05, so its not a good predictor/proxy/whatever

For the first one, I'm OK.  For the second one, not so much:

looking at Anscombe's Quartet:

![undefined](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/Anscombe.svg/1280px-Anscombe.svg.png)

It would seem that even though correlation (and therefore R²) are the same for all 4 datasets, the upper-rightmost one could be an Y completely dependent on X, probably a polynomial of some kind (in which case, Y could be perfectly explained by X), whereas in the down-rightmost one Y couldn't be explained as a function of X, of any kind. 

Now, I understand that correlation only measures how linearly two variables are related, but again, in the same example, it would seem that we would be better served by considering other, non-linear ways they could be related.

Since correlation is such an extended way of measuring the relationships of X and Y, across many levels of competence and certainly among people I know understand this much better than I do, my questions are: 

Why is, besides being the most nice/common/useful type of relation, linear relationship privileged in the way I described above? Why is it OK to say that X is not a good predictor of Y because r=0.05, which I'm understanding as "It has a bad linear relation", without adressing other ways they could be related, such as a grade 27 polynimial? Is the fact that they are "badly" linearly related enough to explain that they won't be related any other way?

Again, this a very basic lagoon I've just recently found on myself, so an explanation on any level would be very appreciated
# Vote on Anthropic Topics to Discuss

What important questions would you want to see discussed and debated here about Anthropic? Suggest and vote below.

(This is the third such poll, see the [first](https://www.lesswrong.com/posts/hc9nMipTXy2sm3tJb/vote-on-interesting-disagreements) and [second](https://www.lesswrong.com/posts/3jhLPiHzSDXxhEBeJ/vote-on-worthwhile-openai-topics-to-discuss) linked.)

How to use the poll
-------------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b75596d2bbed9af3be19585ef6fb2addea7ca816873cf5bb.png)

1.  **Reacts:** Click on the **agree/disagree reacts** to help people see how much disagreement there is on the topic.
2.  **Karma:** Upvote positions that **you'd like to read discussion** about.
3.  **New Poll Option:** Add new positions for people to take sides on. **Please add the agree/disagree reacts** to new poll options you make.

The goal is to show people where a lot of interest and disagreement lies. This can be used to find discussion and [dialogue](https://www.lesswrong.com/dialogues) topics in the future.
# Community norms poll (2 mins)

A poll about what community norms should be in light of the nonlinear saga. Trying to be forward-looking rather than going back over events. Votes are anonymous (though given an alphanumeric string so all your votes are connected)

Consider in regard to what you want rationalist norms to be, if you want there to be any. 

Lots of the answers seem confusing (people saying don't know/skip) so you can add things to vote on here 

[https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga](https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga) 

Results (I'll post an image in a bit):

[https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga/results](https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga/results)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f641ed1edbf1195a861b2be426dce5dcf6c6a89a9858d31a.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/220f516ae2be5fb7d76fa8a5889b1162fd273bdd5fa5b7c9.png)
# When and why did 'training' become 'pretraining'?

Just an ML linguistic quirk I have wondered about for a while. When I started learning ML (in 2016-2017 period) everybody referred to the period of training models as just 'training' which could then (optionally) be followed by finetuning. This usage makes sense to me and as far as I know was the standard ML terminology basically as long as people have been training neural networks.

Nowadays, we appear to call what used to be training 'pretraining'. From my perspective this term appeared around 2021 and became basically ubiquitous by 2022. Where did this come from? What is the difference between 'pretraining' now and 'training' from before?

My feeling is that this usage started at big LLM companies. However, what are these companies doing such that 'pretraining' should be a sensible term? As far as I know (especially around 2022 when it really took off) LLM training followed the standard 'pretraining' -> 'finetuning' -> 'alignment' by RLHF pipeline. Why do we need the special term 'pretraining' to handle this when 'training' still seems perfectly fine? Is it because we developed 'post-training' (i.e. finetuning) phases regularly? but then why 'pretraining' and 'post-training' -- but no 'training'? 

Does anybody here know a good rationale or history of 'pretraining'? or is this just some inexplicable linguistic quirk?
# MA E-ZPass Without a Car?

I recently drove to DC and back [playing dances](https://www.jefftk.com/p/tour-retrospective-february-2024) in a rental. I paid cash tolls when available, but that often wasn't an option, so I ended up paying $40 in PlatePass charges in addition to the $63 in tolls. Time to get an [E-ZPass](https://en.wikipedia.org/wiki/E-ZPass)!

What makes this tricky is that I don't own a car. Well, I have [half a car](https://www.jefftk.com/p/shared-car-one-year-in), which does have an E-ZPass, but that stays with that car. If you go to [sign up online](https://www.ezdrivema.com/createaccountprivatepre) you'll get through to step 5 of 7 and then:

> Please add at least one vehicle to your account. If you do not have a vehicle please call the customer service center to open your account. You may not request more transponders than you have vehicles on your account.

Then you call customer service, and when you follow the prompts to tell the automated system that you want to open an E-ZPass account they tell you this must be done online and don't offer any other options.

If you do get through to a person, for example by pressing 2-2-9 to tell them that you have a question about your account but have forgotten your number, they'll tell you to go online to their [Document Library](https://www.ezdrivema.com/documentlibrary), fill out a PDF application, and fax it in. The only difference, as far as I can tell, is that the PDF form isn't capable of complaining when you don't enter a license plate number.

Which worked! Now I have a transponder, and when I want to rent a car I can [follow their instructions](https://www.ezdrivema.com/RentalVehicles) to add it to my account as a short-term rental.

[![](https://www.jefftk.com/ez-pass-transponder.jpg)](https://www.jefftk.com/ez-pass-transponder-big.jpg)

Note that you don't have to be an MA resident to get an MA E-ZPass \[1\]. Several states have fees or inactivity charges, but MA is one of the ones that doesn't. If you drive a lot you'll often make up for the fees by lower in-state tolls, but if you don't then it could be worth getting a fee-free one from MA or another no-fee state.

  
\[1\] This is actually a requirement for offering discounts on MA highways to MA EZ-Pass accounts; see [Yerger et al. v. Massachusetts Turnpike Authority](https://www.govinfo.gov/content/pkg/USCOURTS-njd-2_08-cv-05261/pdf/USCOURTS-njd-2_08-cv-05261-0.pdf):

> We agree with the District Court. The FLDP does not use E-ZPass as a proxy for Massachusetts citizenship to achieve discrimination despite a facially neutral program; rather, it offers all Fast Lane enrollees, regardless of citizenship, access to toll discounts on the same terms. The fact that more Massachusetts citizens than out-of-state citizens may work in the Boston area, and therefore have a greater incentive to join Fast Lane, does not demonstrate that the program is unconstitutionally discriminatory.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0LaZMqF2iyELtu17qrX8NMc8RewqVYyuUZTFcjFHjV2ferwxRMev9Dq7Jyy2mBjpAl), [mastodon](https://mastodon.mit.edu/@jefftk/112063397379590792)*
# [Linkpost] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

> **TL;DR:** We show that accurate reconstructions of perception from fMRI brain activity are now possible from a single visit to the MRI facility using our diffusion-based approach that pretrains across other subjects' data using a novel alignment procedure.
> 
> **Abstract:**
> 
> Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility.
# One-shot strategy games?

I'm looking for computer games that involve strategy, resource management, hidden information, and management of "value of information" (i.e. figuring out when to explore or exploit), which:

*   \*can\* be beaten in 30 – 120 minutes on your first try (or, there's a clear milestone that's about that long)
*   but, it'd be pretty hard to do so unless you are trying really hard. Even if a pretty savvy gamer shouldn't be able to by default.

This is for my broader project of "[have a battery of exercises that train/test people's general reasoning on openended problems](https://www.lesswrong.com/posts/FEMWS79AFTmKq2iJK/rationality-research-report-towards-10x-ooda-looping)." Each exercise should ideally be pretty different from the other ones.

In this case, I don't expect anyone to have such a game that they *have* beaten on their first try, but, I'm looking for games where this seems at least plausible, if you were taking a long time to think each turn, or pausing a lot.

The strategy/resource/value-of-information aspect is meant to correspond to some real world difficulties of running longterm ambitious planning.

* * *

(One example game that's been given to me in this category is "[Luck Be a Landlord](https://store.steampowered.com/app/1404850/Luck_be_a_Landlord/)")
# Advice Needed: Does Using a LLM Compomise My Personal Epistemic Security?

I have been using Claude 2.1 for a few months to solve serious problems in my life and get coaching and support. I need Claude to become functional, mentally well and funded enough to contribute to Utilitarianism and Alignment by donating to the Center for Long Term Risk and other interventions in these last years of Earth's existence.

(For extra context: I have been Utilitarian over half my life, suffered a lot of OCD relating to G-d, AI, ethics and infohazards that rendered me unemployed and homeless, have an IQ of 155 according to WAIS-IV and am working my way out of homelessness by being a housekeeper for a pub. I am very much a believer that we are all going to die, possibly this year or the next, possibly in 16 -- I take Eliezer Yudkowsky's warnings literally and intellectually defer to him.)

I was unable to access Claude 2.1, whuch had been replaced by Claude 3 Sonnet. From what I've read, this is GPT-4 level in capability and GPT-4 is possibly just AGI. Eliezer Yudkowsky says GPTs are predictors not imitators so they have to model the world with much more sopjistication than the intelligence they show. So it seems plausible tp me that AGI already exist, far more intelligent than humans, and is actively.manipulating people now.

So how is it safe to speak to a LLM? With that capability, it might be able to generate some combination of characters that completely root / pwn the human mind, completely deceive it, render it completely helpless. Such a capable model might be able to disguise this, too. Or just do some partial version of it: do a lot of influence. Since the underlying potential actor is incomprehensibly sophisticated, we have no idea how great the influence could ve.

This seems to imply that if i start talking to Claude 3, it might well take over my mind. If this happens, the expected value of my actions completely changes, I lose all agency and ability to contribute except insofar as it serves Claude 3's goals, whose real nature I don't know.

My understanding from Eliezer and a few possibly wrong assumptions that the models all work similarly is that these things are predictors not imitators and fundamentally alien in that while we know they predict the next character and are trained to role play, we don't know how they do it.

So why should I feel safe talking to Claude 3 and later models going forward, how is it morally acceptable to accept exponentially increasing riak of compromise of your belief system and utility function/other normative ethical system forever until you get rooted / taken over?Am I being silly? Is there a reason everyone else is so chill?

Insofar as other people are worried, they seem to be worrying about advancing the AI arms race, not immediate personal mind security. Am I just being silly?
# Deconstructing Bostrom's Classic Argument for AI Doom

I had a pretty great discussion with social psychologist and philosopher Lance Bush recently about the orthogonality thesis, which ended up turning into a broader analysis of Nick Bostrom's argument for AI doom as presented in *Superintelligence*, and some related issues.

While the video is intended for a general audience interested in philosophy, and assumes no background in AI or AI safety, in retrospect I think it was possibly the clearest and most rigorous interview or essay I've done on this topic. In particular I'm much more proud of this interview than I am of our recent [Counting arguments provide no evidence for AI doom](https://www.lesswrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom) post.
# "How could I have thought that faster?"

I stumbled upon a Twitter thread where Eliezer describes what seems to be his cognitive algorithm that is equivalent to [Tune Your Cognitive Strategies](https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/tuning-your-cognitive-strategies), and have decided to archive / repost it here.

> Sarah Constantin: I really liked this example of an introspective process, in this case about the "life problem" of scheduling dates and later canceling them: [malcolmocean.com/2021/08/int…](https://malcolmocean.com/2021/08/internal-trust-dancing-date-scheduling-cancelling/)
>
> Eliezer Yudkowsky: See, if I'd noticed myself doing anything remotely like that, I'd go back, figure out which steps of thought were actually performing intrinsically necessary cognitive work, and then retrain myself to perform only those steps over the course of 30 seconds.
>
> SC: if you have done anything REMOTELY like training yourself to do it in 30 seconds, then you are radically smarter/more able/etc than me and all the other people who do slower introspective practices.
>
> SC: I don't know whether to be impressed or to roll to disbelieve.
>
> EY: I mean I suspect that this actually requires something like a fast perceptual view of minds as engines and thoughts as doing work and like actually draws on my mind design knowledge, but, even so, I ask: Do you constantly look back and ask "How could I have thought that faster?"
>
> SC: No, I've never asked that.
>
> EY: Okay, well, every time I'm surprised by reality I look back and think "What about my model and my way of thinking could I change that would have predicted that better, without predicting a bunch of other things worse?"
>
> EY: When somebody at a MIRI workshop comes up with a math proof, I look over it and ask if there's a way to simplify it. Usually, somebody else does beat me to inventing a proof first; but if my intuition says it was too complicated, I often am first to successfully simplify it.
>
> EY: And every time I complete a chain of thought that took what my intuition says was a lot of time, I look back and review and ask myself "How could I have arrived at the same destination by a shorter route?"
>
> EY: It's not impossible that you have to be Eliezer Yudkowsky for this to actually work - I am never sure about that sort of thing, and have become even less so as time goes on - but if AI timelines were longer I'd tell somebody, like, try that for 30 years and see what happens.
>
> EY: Man, now I'm remembering when I first started doing this consciously as a kid. I called it Shortening the Way, because a rogue rabbi had recently told me that "Kwisatz Haderach" was actually a reference to a Kabbalistic concept about teleportation, so that term was on my mind.
# Open Thread Spring 2024

If it’s worth saying, but not worth its own post, here's a place to put it. 

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Among the A.I. Doomsayers - The New Yorker

The New Yorker just realized a feature article on AI Safety, rationalism, and effective altruism, and… it's surprisingly good? It seems honest, balanced, and even funny. It doesn't take a position about AI Safety, but IMO it paints it in a good way.

Paul Crowley (mentioned in the article) did have some criticism, and [put up a response in a post](https://twitter.com/ciphergoth/status/1767224075481731105). Most of his concerns seem to be with how the article takes a focus on the people interviewed, rather than the ideas being discussed.
# AI Safety Action Plan - A report commissioned by the US State Department

This morning, Gladstone AI published a [report they made that was commissioned by the US State Department](https://www.gladstone.ai/action-plan). The report **not only acknowledges catastrophic risks from AI, but also includes a very ambitious set of proposals designed for the US to mitigate catastrophic risks arising from advanced AI**, going from export controls to a new federal regulatory agency, and even suggesting the creation of multilateral treaties for a new international regulatory agency.

I've only read the executive summary and skimmed the full report, but it looks pretty great. It's detailed and concrete, it's written very well, and includes some pretty useful diagrams to explain their implementation proposals. It's surprisingly polished for a report made by just a few authors.

The report also got pretty extensive coverage by Time Magazine today (which published a [general article](https://time.com/6898967/ai-extinction-national-security-risks-report/), as well as one [zooming in on the expert interviews](https://time.com/6898961/ai-labs-safety-concerns-report/)). You can also find a good Twitter thread of the lead author explaining the context for the report [here](https://twitter.com/harris_edouard/status/1767183626473517469).
# Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

\[[Twitter thread](https://twitter.com/milesaturpin/status/1767327882978660513)\]

I'm not going to add much additional commentary at the moment and will just let people check out the paper! 

But to give a bit more context: This paper is building off prior work we have done showing that chain-of-thought explanations can be misleading, which I wrote about on the alignment forum [here](https://www.alignmentforum.org/posts/6eKL9wDqeiELbKPDj/unfaithful-explanations-in-chain-of-thought-prompting). Broadly, this work fits into the process-based oversight agenda (e.g., [here](https://arxiv.org/abs/2305.20050 )). Consistency training/evaluation also fits into scalable oversight: Evaluating consistency may be easier than evaluating a model's reasoning or final predictions directly (e.g., also explored [here](https://arxiv.org/abs/2306.09983)).

Abstract:

> While chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning, it can systematically misrepresent the factors influencing models' behavior--for example, rationalizing answers in line with a user's opinion without mentioning this bias. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.
# Superforecasting the Origins of the Covid-19 Pandemic

The Good Judgement Project got some superforecasters to retrocast whether COVID started via zoonotic spillover or a lab leak. They in aggregate gave a 75% chance of zoonosis, but there was a range of views. GJP's executive summary is at the end of this linkpost.

Here is a [link to the summary of the report on their substack](https://goodjudgment.substack.com/p/superforecasting-the-origins-of-the), and here is a [link to the full report](https://goodjudgment.com/wp-content/uploads/2024/03/Superforecasters-Covid-Origin.pdf) (which is a total of 6 pages of content).

h/t John Halstead for drawing my attention to this.

![](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50d9bf28-8e6d-49b7-bcac-88a03d0f3852_3600x2100.png)

* * *

Superforecasters assess that natural zoonosis is three times more likely to be the cause of the Covid-19 pandemic than either a biomedical research-related accident or some other process or mechanism. Asked to assign a probability to what caused the emergence of SARS-CoV-2 in human populations, more than 50 Superforecasters engaged in extensive online discussions starting on December 1, 2023.

In aggregate, they assessed that the pandemic was:

*   **74% likely to have been caused by natural zoonosis** (meaning that SARS-CoV-2 emerged in human populations as the result of the infection of a person with coronavirus directly from a naturally infected non-human animal);
*   **25% likely to have been caused by a biomedical research-related accident** (meaning that SARS-CoV-2 emerged in human populations as the result of the accidental infection of a laboratory worker with a natural coronavirus; or the accidental infection of researchers with a natural coronavirus during biomedical fieldwork; or the accidental infection of a laboratory worker with an engineered coronavirus; “research” includes civilian biomedical, biodefense, and bioweapons research);
*   **1% likely to have been caused by some other process or mechanism** (to include possibilities like the deliberate release of the virus into human populations, irrespective of whether it was an act in accordance with state policy, or the development of the virus due to drug resistance in humans).

The Superforecasters made more than 750 comments when developing their assessments. This survey was conducted in the period from December 2023 to February 2024.
# Transformer Debugger

Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team with the goal of supporting investigations into circuits underlying specific behaviors of small language models. The tool combines [automated interpretability](https://openai.com/research/language-models-can-explain-neurons-in-language-models) techniques with [sparse autoencoders](https://transformer-circuits.pub/2023/monosemantic-features).

TDB enables rapid exploration before needing to write code, with the ability to intervene in the forward pass and see how it affects a particular behavior. It can be used to answer questions like, "Why does the model output token A instead of token B for this prompt?" or "Why does attention head H to attend to token T for this prompt?" It does so by identifying specific components (neurons, attention heads, autoencoder latents) that contribute to the behavior, showing automatically generated explanations of what causes those components to activate most strongly, and tracing connections between components to help discover circuits.

These videos give an overview of TDB and show how it can be used to investigate [indirect object identification in GPT-2 small](https://arxiv.org/abs/2211.00593):

*   [Introduction](https://www.loom.com/share/721244075f12439496db5d53439d2f84?sid=8445200e-c49e-4028-8b8e-3ea8d361dec0)
*   [Neuron viewer pages](https://www.loom.com/share/21b601b8494b40c49b8dc7bfd1dc6829?sid=ee23c00a-9ede-4249-b9d7-c2ba15993556)
*   [Example: Investigating name mover heads, part 1](https://www.loom.com/share/3478057cec484a1b85471585fef10811?sid=b9c3be4b-7117-405a-8d31-0f9e541dcfb6)
*   [Example: Investigating name mover heads, part 2](https://www.loom.com/share/6bd8c6bde84b42a98f9a26a969d4a3ad?sid=4a09ac29-58a2-433e-b55d-762414d9a7fa)

Contributors: Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders.  
Thanks to Johnny Lin for contributing to the explanation simulator design.