Harley-ml/lesswrong huggingface dataset:
# What can we learn from insecure domains?

Cryptocurrency is terrible.  With a single click of a button, it is possible to accidentally lose *all of your funds*.  99.9% of all cryptocurrency projects are complete scams (conservative estimate).  Crypto is also tailor-made for ransomware attacks, since it makes it possible to send money in such a way that the receiver has perfect anonymity.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9a9c7ee1dcb0af0de861e8a442eeba8edd0e3a16b5a59474.png)

typical web3 experience

Similarly, Cyber Security is terrible.  Basically every computer on the internet is infected with multiple types of malware.  If you have ever owned a web-server with a public IPV4 address, you undoubtedly have had the pleasure of viewing a log file that looks like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a47c19de49d66ae1e0498bfd27aeeffce6a1479991faf97a.png)

Every IPV4 address on earth is under constant attack by malware

In a few months, the world is about to be introduced to a brand new *insecure by design* platform, the LLM agent:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d28c052160932d809f5533e3bc35a18c4bda160574a6cac8.png)

No one worth taking seriously believes that Microsoft Copilot (or Anthropic, or any other LLM agent) is going to be remotely secure against prompt injection attacks.

One fascinating thing (to me) about these examples is that they all basically work fine[^yfe0qe9bmqh].  Despite being completely broken, normal people with normal intelligence use these systems routinely without losing 100% of their funds. This happens despite the fact that people with above-average intelligence have a financial incentive to take advantage of these security flaws.  

One possible conclusion is along the lines of "everything humanity has ever built is constantly on fire.  We must never built something existentially dangerous or we're already dead."

However we already did:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6bfab63af632b95b0a544f2fec756bc82a9fff478e425a12.jpg)

And like everything else, the story of nuclear weapons is that they are [horribly insecure](https://en.wikipedia.org/wiki/1961_Goldsboro_B-52_crash) and [error prone](https://en.wikipedia.org/wiki/Stanislav_Petrov).

What I want to know is *why*?  Why is it that all of these systems, despite being hideously error prone and blatantly insecure by design somehow still work?

I consider each of these systems (and many like them) a sort of standing challenge to the fragile world hypothesis.  If the world is *so* fragile, why does it keep *not* ending?

[^yfe0qe9bmqh]: If anyone would like to make a bet, I predict 2 years from now LLM agents:will be vulnerable to nearly trivial forms of prompt-injectionMillions of people will use them to do things like spend money that common-sense tells you not to do on a platform this insecure by design
# The Median Researcher Problem

Claim: memeticity in a scientific field is mostly determined, not by the most competent researchers in the field, but instead by roughly-median researchers. We’ll call this the “median researcher problem”.

Prototypical example: imagine a scientific field in which the large majority of practitioners have a very poor understanding of statistics, p-hacking, etc. Then lots of work in that field will be highly memetic despite trash statistics, blatant p-hacking, etc. Sure, the *most* competent people in the field may recognize the problems, but the median researchers don’t, and in aggregate it’s mostly the median researchers who spread the memes.

(Defending that claim isn’t really the main focus of this post, but a couple pieces of legible evidence which are weakly in favor:

*   People did in fact try to [sound the alarm about poor statistical practices well before the replication crisis](https://journals.plos.org/plosmedicine/article/file?id=10.1371%2Fjournal.pmed.0020124&type=printable), and yet practices did not change, so clearly at least some people did in fact see the problem and were in fact not memetically successful at the time. The claim is more general than just *statistics*-competence and replication, but at least in the case of the replication crisis it seems like the model must be at least somewhat true.
*   Again using the replication crisis as an example, you [may have noticed](https://archive.thetab.com/us/2017/04/10/which-major-has-highest-iq-64811) the very wide (like, 1 sd or more) average IQ gap between students in most fields which turned out to have terrible replication rates and most fields which turned out to have fine replication rates.

… mostly, though, the reason *I* believe the claim is from seeing how people in fact interact with research and decide to spread it.)

Two interesting implications of the median researcher problem:

*   A small research community of unusually smart/competent/well-informed people can relatively-easily outperform a whole field, by having better internal memetic selection pressures.
*   … and even when that does happen, the broader field will mostly not recognize it; the higher-quality memes within the small community are still not very fit in the broader field.

In particular, LessWrong sure seems like such a community. We have a user base with probably-unusually-high intelligence, community norms which require basically everyone to be familiar with statistics and economics, we have fuzzier community norms explicitly intended to avoid various forms of predictable stupidity, and we definitely have our own internal meme population. It’s exactly the sort of community which can potentially outperform whole large fields, because of the median researcher problem. On the other hand, that does not mean that those fields are going to recognize LessWrong as a thought-leader or whatever.
# Goal: Understand Intelligence

Save the world by understanding intelligence.

Instead of having SGD "grow" intelligence, design the algorithms of intelligence directly to get a system we can reason about. Align this system to a narrow but pivotal task, e.g. upload a human.

The key to intelligence is finding the algorithms that infer world models that enable efficient prediction, planning, and meaningfully combining existing knowledge.

By understanding the algorithms, we can make the system non-self-modifying (algorithms are constant, only the world model changes), making reasoning about the system easier.

Understanding intelligence at the algorithmic level is a very hard technical problem. However, we are pretty sure it is solvable and, if solved, would likely save the world.

Current focus: How to model a world such that we can extract structure from the transitions between states ('grab object'=useful high level action), as well as the structure within particular states ('tree'=useful concept).

I am leading a project on that. Read more [here](https://docs.google.com/document/d/1I8cWFNO5FHFZVVpgX6yn_l3hXw8dzVtiFuDlyYqBRJQ/edit?tab=t.0#heading=h.z0rukj2zv4a4) and apply [on the AISC website](https://www.aisafety.camp/).
# [Linkpost] Building Altruistic and Moral AI Agent with
Brain-inspired Affective Empathy Mechanisms

Abstract

> As AI closely interacts with human society, it is crucial to ensure that its decision-making is safe, altruistic, and aligned with human ethical and moral values. However, existing research on embedding ethical and moral considerations into AI remains insufficient, and previous external constraints based on principles and rules are inadequate to provide AI with longterm stability and generalization capabilities. In contrast, the intrinsic altruistic motivation based on empathy is more willing, spontaneous, and robust. Therefore, this paper is dedicated to autonomously driving intelligent agents to acquire morally behaviors through human-like affective empathy mechanisms. **We draw inspiration from the neural mechanism of human brain’s moral intuitive decision-making, and simulate the mirror neuron system to construct a brain-inspired affective empathydriven altruistic decision-making model. Here, empathy directly impacts dopamine release to form intrinsic altruistic motivation. Based on the principle of moral utilitarianism, we design the moral reward function that integrates intrinsic empathy and extrinsic self-task goals.** A comprehensive experimental scenario incorporating empathetic processes, personal objectives, and altruistic goals is developed. **The proposed model enables the agent to make consistent moral decisions (prioritizing altruism) by balancing self-interest with the well-being of others.** We further introduce inhibitory neurons to regulate different levels of empathy and verify the positive correlation between empathy levels and altruistic preferences, yielding conclusions consistent with findings from psychological behavioral experiments. This work provides a feasible solution for the development of ethical AI by leveraging the intrinsic human-like empathy mechanisms, and contributes to the harmonious coexistence between humans and AI. \[emphasis mine\]

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e35cb3e9264451ab8b0ab7fa2e8d5fc2121d44741f9ac1b4.png)
# Does the "ancient wisdom" argument have any validity? If a particular teaching or tradition is old, to what extent does this make it more trustworthy?

Proponents of spirituality and alternative medicine often use the argument "this has been practiced for 2000 years", with the subtext "therefore it must work". Does this argument have any validity?

At first glance I want to reject the argument entirely, but that might be premature. Are there situations where this kind of argument is valid or somewhat valid?

I was reminded of this question when I read Shaila Catherine's book *The Jhanas* (about certain ecstatic meditation states mentioned in Buddhism) and she said something like: "Trust in the method. Buddhists have been practicing it for 2600 years. It works. Your mind is not the exception." This argument did not seem valid to me, because AFAIK Buddhist monasteries do not publish records of how many of their monks achieve which states and insights - *au contraire*, I believe monks have a taboo against talking about their attainments. So I know of no evidence that most practitioners can achieve jhana. From what I know, it is entirely plausible that only a small fraction of practitioners ever succeed at these instructions, and that therefore their minds are the exception, not mine.
# Update on the Mysterious Trump Buyers on Polymarket

I've written a follow-up post on the mysterious Trump buyers on Polymarket. While mainstream media has extensively covered this story, it has overlooked some critical details—most notably, that this trader's bet on Trump is closer to $75 million USDC, making it the largest election market wager to date. Regardless of the outcome, Theo is poised to go down in history as the most significant bettor in prediction markets.

[Link to Post](https://jorgevelez.substack.com/p/the-mysterious-trump-buyers-on-polymarket-2)
# Bay Winter Solstice 2024: Speech Auditions

Hello! I'm looking for community members to read speeches at the Bay Winter Solstice event this year. If you're interested, please email me at ozybrennan@gmail.com by the end of **November 17** with the speeches you're potentially interested in. I'll ask you to record yourself reading it (no need to be particularly polished).   

I am currently auditioning for:

*   [Thanksgiving Prayer](https://www.lesswrong.com/posts/QB6BkkpwiecfF6Ekq/thanksgiving-prayer)
*   [I Have Seen The Tops Of Clouds](https://medium.com/message/i-have-seen-the-tops-of-clouds-e21c5941de32)*
*   [Self-Compassion](https://mindingourway.com/self-compassion/)*
*   [What Resembles The Grave But Isn't](https://billmoyers.com/story/poetry-month-what-resembles-the-grave-but-isnt/)
*   [500 Million But Not A Single One More](https://forum.effectivealtruism.org/posts/jk7A3NMdbxp65kcJJ/500-million-but-not-a-single-one-more)
*   [RMS Carpathia](https://mylordshesacactus.tumblr.com/post/170401018158/please-make-a-post-about-the-story-of-the-rms)
*   [The Gift We Give To Tomorrow](https://www.lesswrong.com/posts/pGvyqAQw6yqTjpKf4/the-gift-we-give-to-tomorrow)*
*   An original piece about the malaria vaccine
*   An original piece about the abolition of slavery

(The list is tentative and does not include all speeches at Solstice.)

For starred items, I am only including an excerpt of the speech, so please email me to get the excerpt I'm using before you record! 

As a reminder, the [event itself](https://www.eventbrite.com/e/bay-area-winter-solstice-2024-tickets-958115718907) will be the evening of **Friday, December 20** in Berkeley. There will also be a dress rehearsal on the evening of **Tuesday, December 17** in Berkeley, which you should plan on coming to if you're leading a song or speech.
# Lighthaven Sequences Reading Group #10 (Tuesday 11/12)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [What is Evidence?](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence)
2.  [How Much Evidence Does It Take?](https://www.lesswrong.com/posts/nj8JKFoLSMEmD3RGp/how-much-evidence-does-it-take)
3.  [Conservation of Expected Evidence](https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence)
4.  [Einstein's Arrogance](https://www.lesswrong.com/posts/MwQRucYo6BZZwjKE7/einstein-s-arrogance)
5.  Optional: [Mistakes with Conservation of Expected Evidence](https://www.lesswrong.com/posts/zTfSXQracE7TW8x4w/mistakes-with-conservation-of-expected-evidence)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

Doors open 6pm (yes we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like. If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# How to put California and Texas on the campaign trail!

In the USA, the president isn't determined by a straight vote. Instead, each state gets a certain number of Electoral College (EC) votes, and the candidate with 270 EC votes wins.

It's up to each state to decide how to allocate its EC votes. Most do “winner-takes-all,” but some, e.g., Maine and Nebraska, split them up.

California and Texas have the most EC votes of any state, with 54 and 40 votes respectively, so you would think they would get a lot of love from presidential candidates. Instead, they're mostly ignored—California will always be Blue, and Texas Red, so what's the point of pandering to them? This is clearly bad for Californians and Texans as their interests aren't listened to.

So why doesn't California switch to a proportional EC vote split? One that would ensure that there's always something to gain by pandering to Californian interests?

Because most Californians are Democrats, and while a proportional vote split would be good for Californians, it would be bad for Democrats, who in the 2020 election would have lost 20 EC votes—over half of Biden's margin of 37. All in all, not worth it.

But the same issue impacts Texas too, which, if they'd switched to proportional splitting, would have handed Biden 18 votes.

This suggests an obvious solution: California and Texas should mutually agree to split their vote proportionally. That way, neither Republicans nor Democrats significantly gain, but California and Texas would finally be on the presidential campaign circuit!
# How to cite LessWrong as an academic source?

Hi there.

Quick question. I am using a few articles from LessWrong for a dissertation. Are there any mainstream articles/sources that reference LessWrong as being the catalyst/partial source for AI alignment, researchers, and other academic literature? I think it's snobbish, or, discriminatory to regard LessWrong as merely another online website. I was hoping to get some advice on how to formulate a paragraph justifying the citation of LessWrong?

Thanks.
# Scissors Statements for President?

(Epistemic status: I spoke simply / without "appears to" hedges, but I'm not sure of this at all.)

I’m confused why we keep getting [scissors](https://en.m.wiktionary.org/wiki/scissor_statement) [statements](https://slatestarcodex.com/2018/10/30/sort-by-controversial/) as our Presidential candidates, but we do.  (That is: the candidates seem to break many minds/communities.)

A toy model:[^lxlh1ailrx]

Take two capacities, *A* and *B.  *Ideally anti-correlated.

Craft two candidates:

*   Candidate *X*, who seems acceptable if you’re *A*-blind (if you have a major gap in your situation awareness near *A*).
*   Candidate *Y*, who seems acceptable if you’re *B*-blind (if you have a major gap in your situation awareness near *B*).

Now let voters talk.

“How can you possibly vote for *X*, given how it’ll make a disaster on axis *A*?”, asks Susan.  (She is *B*-blind, which is part of why she is so confused/irate/loud here.)  Susan inquires in detail.  She (accurately) determines the staunchest *X*-voters don't understand *A*, and (understandably, but incorrectly) concludes that this explains their *X*-voting, that they have nothing to teach her, and that she should despair of working well with anyone who voted for Candidate *X*.

““How can you possibly vote for *Y*, given how it’ll make a disaster on axis *B*?”, asks Robert.  He, too, inquires in detail.  And he (accurately) determines the staunchest *Y*-voters have a key basic blind spot where he and his friends/neighbors have sense... feels a sense of closure ("okay, it's *not* that they know something I don't know"), and despairs of working well with anyone who voted for *Y*.

The thing that annoys me about this process is that, in the wake, it is harder for both sets of voters to heal their own blind spots.  “Being able to see A accurately” is now [linked up socially and verbally](https://www.lesswrong.com/tag/bucket-errors) with “being one of the people who refuse to acknowledge B” (and vice versa).  (This happens because the ontology has been seized by the scissors-statement crafters – there is a common, salient, short word that means *both* “*A* matters” and “*B* is fake,” and people end up using it in their own head, and, while verifying a real truth they can see, locking in a blind spot they can’t see.)

[^lxlh1ailrx]: This is a toy model for how the "scissors-ness" works, not for why some process is crafting us candidates like that.  I don't have a guess about that part.  Though I like these articles.
# Anthropic: Three Sketches of ASL-4 Safety Case Components

> The cleanest argument that current-day AI models will not cause a catastrophe is probably that they lack the capability to do so.  However, as capabilities improve, we’ll need new tools for ensuring that AI models won’t cause a catastrophe *even if we can’t rule out the capability*. Anthropic’s [Responsible Scaling Policy](https://www-cdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf) (RSP) categorizes levels of risk of AI systems into different AI Safety Levels (ASL), and each level has associated commitments aimed at mitigating the risks. Some of these commitments take the form of *affirmative safety cases*, which are structured arguments that the system is safe to deploy in a given environment. Unfortunately, it is not yet obvious how to make a safety case to rule out certain threats that arise once AIs have sophisticated strategic abilities. The goal of this post is to present some candidates for what such a safety case might look like.

This is a post by Roger Grosse on Anthropic's new Alignment Science Blog. The post is *full* of disclaimers about how it isn't an official plan and doesn't speak for the org (and that it's inadequate: "none of the sketches presented here fully succeeds in addressing the sabotage risk"). But presumably it's Anthropic's best *sketch of ASL-4 safety cases*.

The three safety cases are Mechanistic Interpretability, AI Control, and Incentives Analysis.

Regardless of how good these safety cases are, it's good when labs share their thinking on safety stuff; yay Anthropic.
# New Funding Category Open in Foresight's AI Safety Grants

[Foresight Institute](https://foresight.org)'s AI Safety Grants Program added a new focus area in response to the continually evolving field. Moving forward, our funding ($1.5M-$2M annually) will be allocated across the following four focus areas:

1.  **Automating AI-relevant research and forecasting**

*   Scaling AI-enabled research to support safe AGI development
*   Scaling efficient forecasting methods relevant for safe AGI
*   Other approaches in this area

**2\. Neurotech to integrate with or compete against AGI**

*   Brain Computer Interfaces (BCI) to enhance human cognition or facilitate human-AGI collaboration
*   Whole Brain Emulations (WBE) which might function as human-like general intelligences that are more interpretable and alignable than AGI
*   Lo-fi emulations using behavioral and neural data with deep learning, potentially offering a cost-effective alternative to full WBEs
*   Other approaches in this area

**3\. Security technologies for securing AI systems**

*   Implementations of computer security techniques (including POLA, SeL4-inspired systems, and hardened hardware security) to safeguard AI systems
*   Automated red-teaming for AI security and capabilities
*   Cryptographic and related techniques to enable trustworthy coordination architectures
*   Other concrete approaches in this area

**4\. Safe multipolar human AI scenarios**

*   Game theory that addresses interactions between multiple humans, AIs, or ultimate AGIs
*   Avoiding collusion and deception and/or encouraging pareto-preferred/positive-sum dynamics
*   Approaches for addressing principal-agent problems in multi-agent systems
*   Other concrete approaches in this area

### **Application Process**

We accept applications on a quarterly cycle, with deadlines at the end of March, June, September, and December. Decisions are made within 8 weeks of each deadline.

**Next Deadline**: December 31st, 2024.

For more information and to apply: [https://foresight.org/ai-safety](https://foresight.org/ai-safety)
# Should CA, TX, OK, and LA merge into a giant swing state, just for elections?

As Americans know, the electoral college gives disproportionate influence to swing states, which means a vote in the extremely blue state of California was basically wasted in the 2024 election, as are votes in extremely red states like Texas, Oklahoma, and Louisiana. State legislatures have the Constitutional power to assign their state's electoral votes. So why don't the four states sign a compact to assign all their electoral votes in 2028 and future presidential elections to the winner of the aggregate popular vote in those four states? Would this even be legal?

The population of CA is 39.0M (54 electoral votes), and the population of the three red states is 38.6M (55 electoral votes). The combined bloc would control a massive 109 electoral votes, and would have gone for Biden in 2020 and Trump in 2024. Every state has an incentive to sign to increase their voters' influence in the national election.

There has been one similar proposal before: a [National Popular Vote Interstate Compact](https://en.wikipedia.org/wiki/National_Popular_Vote_Interstate_Compact) which would go into effect when states controlling >50% of electoral votes sign it. It is only popular with blue states-- swing states don't want to reduce their influence, and red states don't want to give up the Republican electoral college advantage. Merging states would be far superior incentive-wise, as the influence of every signatory would increase and there would be essentially no expected net shift in the election results.

The compact should only go into effect once all 4 states sign, of course. But even then, there is another potential problem: Say that polls suggest the combined bloc is trending blue in 2028. The Louisiana legislature has an incentive to pull out at the last minute, and the other red states will follow. To prevent this, there must be a provision that once signed by all 4 states, the compact can't be repealed by any state until after the next election.

Smaller states have an even greater incentive to merge, e.g. Rhode Island with Montana. With only two states in the compact, there are fewer difficulties in getting all the states to sign at once. And due to their slightly higher electoral vote counts per voter, these small states' voters would quickly go from among the least important in the election to the most important.
# Signaling with Small Orange Diamonds

There are so many important efforts to make the world better that are significantly limited by funding, and it would be great if we could have a culture where significant and thoughtful giving was normal and common. It's hard to build that sort of norm if people keep their giving private, however, and so I've [long been](https://www.jefftk.com/p/make-your-giving-public) an advocate of being public about your giving. I [list my donations](https://www.jefftk.com/donations) (jointly with [Julia](https://juliawise.net/)) and have taken [Giving What We Can](https://www.givingwhatwecan.org)'s [10% Pledge](https://www.givingwhatwecan.org/pledge) (also jointly with Julia).

In July [GWWC suggested](https://forum.effectivealtruism.org/posts/uZzXRyAwkDHLfu94W/we-ve-renamed-the-giving-what-we-can-pledge) people put the "small orange diamond" symbol (🔸) in their usernames on social media to show that they've pledged. Here's how the EA Forum describes this on the profile editing page:

[![](https://www.jefftk.com/ea-forum-edit-profile-diamond.png)](https://www.jefftk.com/ea-forum-edit-profile-diamond-big.png)

This digital symbol reminds me of the physical [Symbolic Beads of Raikoth](https://slatestarcodex.com/2013/05/11/raikoth-symbolic-beads/). In an older Scott Alexander post he talked about how his fictional society attempted to redirect humanity's natural competitive status-signaling in a more productive direction than yachts. The symbol also has something in common with wedding rings, showing that you have taken on a serious commitment. To the extent that it helps promote a norm of substantial and effective giving, that seems pretty good!

And yet despite being on the board of GWWC USA I haven't put it in my username, even on the EA Forum where it would be most relevant. I'm not sure if this is the right call, but some things pushing me in this direction:

*   Usernames with symbols in them feel like they're signaling something I don't want to signal, just by the inclusion of emoji. Something like "I'm a very online person who keeps up with fast-moving discourse".
    
*   Relatedly, it feels like this is not what the username field is for. If I'm interacting with someone on some topic unrelated to my advocacy it feels intrusive and uncooperative to be bringing it into the conversation.
    
*   While effective giving is one thing I would like to see more of, this is really a large category. I could see including symbols showing that I'm an advocate for allowing people to build housing, giving kids more independence, applying your career effectively, increasing immigration, [etc](https://www.jefftk.com/p/policies). But I don't want to be "Jeff Kaufman 🔸🏗👣🛝💡🌎".
    

For now I've decided I will go ahead and add this to my name on the EA Forum where it's most relevant and I most understand how it will be perceived, but I won't add it to my username elsewhere. If you'd like to try to convince me to do otherwise, please go ahead!
# Analyzing how SAE features evolve across a forward pass

*This research was completed for the Supervised Program for Alignment Research (SPAR) summer 2024 iteration. The team was supervised by *[*@Stefan Heimersheim*](https://www.lesswrong.com/users/stefan42) *(Apollo Research). Find out more about the program and upcoming iterations*[*here*](https://sparai.org/)*.*

TL,DR: We look for related SAE features, purely based on statistical correlations. We consider this a cheap method to estimate e.g. how many new features there are in a layer and how many features are passed through from previous layers (similar to the feature lifecycle in Anthropic’s [Crosscoders](https://transformer-circuits.pub/2024/crosscoders/index.html#motivating-ex-2)). We find communities of related features, and features that appear to be quasi-boolean combinations of previous features.

Here’s a [web interface](https://stefanhex.com/spar-2024/feature-browser/) showcasing our feature graphs.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXen5AnWBTq-7_zj80s0uYZKxagAe5kZ_WzZPTUukHZ2DlYSd7fYRdRty-oAMCOJl62zK57rmlpRPqz-bivYXMG8UFNAzN2ilsTBhldXzUil6qFbzJ_YVqqu_AdMxzgkYIs1nO6EyBEbDq4n6-qHq_a36tQ?key=3ZRJqqrN2N12IMYPpCVIXQ)

**Communities of sparse features through a forward pass**. Nodes represent residual stream SAE features that were active in the residual stream for a specific prompt of text. The rows of the graph correspond to layers in GPT-2 (the bottom row is an earlier layer). The edges represent the Jaccard similarity of the activations of features across many other prompts. Colors represent different subgraphs discovered by a community-finding algorithm. Features within a community typically capture similar concepts. More graphs can be viewed in the [feature browser](https://stefanhex.com/spar-2024/feature-browser/).

We ran many prompts through GPT-2 residual stream SAEs and measured which features fired together, and then created connected graphs of frequently co-firing features (“communities”) that spanned multiple layers (inspired by work from [Marks 2024](https://arxiv.org/abs/2403.19647)). As expected, features within communities fire on similar input tokens. In some circumstances features appeared to “specialize” in later layers.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1cc81df902ad1803757afe2fc12d996b580effc9a20dcee1.png)

**Feature evolution**: a layer 2 SAE feature (bottom) which detects “evidence” in many different contexts specializes into later layer features which detect “evidence” in mutually exclusive contexts. These relationships were discovered by counting, of all the times that a feature activated on a set of inputs, which previous-layer feature activated most frequently as well.

You can read the paper on [arXiv](https://arxiv.org/abs/2410.08869), or find us at the [ATTRIB](https://attrib-workshop.cc/) workshop at NeurIPS!
# Boring & straightforward trauma explanation

*I thought of this a couple years ago and figured it was so obvious that it wasn't worth posting about, but people are still discussing trauma endlessly, and I have not seen an explanation written anywhere, so here's this.*

"Trauma" is a bad experience deemed anomalous. It means "the world is not *usually* like that". We do not call any behavior or emotional pattern "trauma" if it is obviously adaptive.

If you are in a war and you lie down and panic when you hear loud bangs, nobody will call you traumatized. When the war is over and you panic at fireworks, people will say you are traumatized. If then a bomb lands nearby and you survive because you took cover, they'll say you were smart and acted fast.

If 10% of your country got murdered a generation or two ago for not going with the political majority, then you are completely sane for shutting off the thinky brain in politics. (The USSR, China, Korea, Vietnam, Nigeria, Sudan, Afghanistan, Cambodia, Ethiopia, Lebanon...) If that won't happen where you live now, it's "intergenerational trauma".

If you got raped (and half your friends did too) and you are anxious and distrustful, then maybe you are just correctly calibrated about your own social sphere. If it was in a different country 10 years ago, then it's trauma.

If you got ostracized and called creepy in school, you might have a good idea of what actually happens when you ask girls out in public. If your classmates were jerks and you don't have acne anymore, it's trauma.

If you got emotionally beat up by all your exes, then being less open might be a good idea. If they were all cocaine addicts from the same town, it's trauma.

If driving a car makes you want to throw up since your last accident, you might be a bad driver. If a helicopter landed on you, it's trauma. (I have heard it said that if you're worried about your driving then you're probably a safe driver. The last person I met who was seriously worried about their driving totaled her car a week later. Everybody, including me, was telling her she was fine and shouldn't worry.)

It often takes years to get over trauma because *that's how long it takes to get enough evidence.* Even a terrible driver will only crash every few years. And you can't trust your friends (or therapist) to make a very accurate assessment of the risk, because nobody has good data.
# The King and the Golem - The Animation

Rational Animations' new video is an animation of [The King and the Golem](https://www.lesswrong.com/posts/bteq4hMW2hqtKE49d/the-king-and-the-golem), by [@Richard_Ngo](https://www.lesswrong.com/users/ricraz?mention=user), with minimal changes to the original text (we removed some dialogue tags). I hope you'll enjoy it!
# New UChicago Rationality Group

Hey y'all! 

I just started a rationality group on the UChicago campus and wanted to post it here to advertise it to UChicago-affiliated LessWrong readers. We've had a couple meetings so far which have been great, and I'm excited for more! A few more things: 

(1) You can join the email list by submitting [this](https://forms.gle/mbhY37Vo9XHN4vEU7) form to be updated about meeting times, hear about events, and more! 

(2) While we do not yet have university RSO (Registered Student Organization) status, I plan on applying  for fall of next year. This would allow us to promote the club at the annual club fair, put up posters around campus (for events, fellowships, or other opportunities), and get some funding from the university. 

(3) In general, the purpose of this group is threefold: (1) to create an in person community for those who would consider themselves rationalists (in the LW/ other rationalist blogosphere sense), (2) to work on the martial art of rationality (for example, I plan to run a forecasting fellowship at some point), and (3) to counterfactually introduce UChicago students and faculty to rationalist ideas and the rationality community (both to raise the sanity waterline and increase the number of Effective Altruists on campus). 

Any comments, questions, or suggestions would be greatly appreciated. If you want to reach out without using the comment section, you can also email me at noahdbirnbaum@gmail.com.
# GPT-4o Can In Some Cases Solve Moderately Complicated Captchas

Here are several examples; I found these captchas via the web rather than generating them anew, but none of them came attached to solutions so I'm not sure their presence in the training data would affect things in any case. (That said, it's possible that the lower resolution of the latter two degraded the adversarial perturbation; I would appreciate a source of higher-resolution captchas if anyone happens to know one.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/23d75eaf88c4279d0b0bfbfd78c521e8210685dcc08d747a.png)

It clearly couldn't see all the objects, but the owl was in fact the correct answer

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/884317656dc62c25cc5d2ddea8583d1cd00fa1b2b94f73a3.png)

Entertaining failure at basic numerals while nonetheless answering correctly here

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7be443826492ac16d7d3c6b68f1fdaba77a46e7007eb19d6.png)

This one I was surprised by; I expected the image to be too low-resolution to be comprehensible, but 8/9 are correct here (the middle left image is a chair with an unusually low back)
# How should vegans think about Methionine needs?

Non-animal based protein sources mostly have a different amino acid profile than animal-based protein sources.  Different plants also have a different composition. 

From looking a bit at the data myself it seems that if you mix different plant protein sources, you can get a good balance for most amino acids with the expectation of Methionine.

[Mike from Renaissance Periodization](https://www.youtube.com/watch?v=MB7rIAArV2Q) who's a professor of exercise science suggests that a vegan can just look at the Protein digestibility-corrected amino acid score (PDCAAS) and use this as a factor for protein consumption to get the correct amino acid consumption as a vegan. 

From thinking about the issue myself, I would expect that you get significantly lower Methionine consumption if you do that then a person who uses animal-based products as their protein sources.

Given that a protein needs a very exact number of each amino acid to be synthesized, for essential amino acids like Methionine, I would expect their consumptions to be a bottleneck for muscle building which needs protein. Even if all the other proteins are in good amounts and thus the PDCAAS score is decent, you can drown in a river that's on average 20cm deep.

One possible solution would be to focus on high Methionine protein sources as a vegan and less on the PDCAAS (for a vegan protein source, soy is good at both) and just consume twice the amount of protein that a non-vegan would to get similar Methionine consumption. I'm not sure what the exact consequences of having all the other amino acids in excess happen to be.

Has someone thought more about this and come to a good conclusion about how to think about Methionine as someone with a mostly vegan diet who wants to build muscle?
# Bay Winter Solstice 2024: song leading auditions

Hello! I am looking for community members to lead songs at the Bay Winter Solstice this year. (I am the music director; Ozy is the overall creative lead.)

If you're interested, please send me your audition recordings by the end of **November 24** via this form: [https://docs.google.com/forms/d/e/1FAIpQLSegrNKXJCTS69ioGUxSaMDXqz60X5zyb6kBDCuTVcTF6Mu3Ig/viewform](https://docs.google.com/forms/d/e/1FAIpQLSegrNKXJCTS69ioGUxSaMDXqz60X5zyb6kBDCuTVcTF6Mu3Ig/viewform)

The form contains audition instructions, as well as a list of songs we're currently auditioning for.

As a reminder, the [event itself](https://www.eventbrite.com/e/bay-area-winter-solstice-2024-tickets-958115718907) will be the evening of **Friday, December 20** in Berkeley. There will also be a dress rehearsal on the evening of **Tuesday, December 17** in Berkeley, which you should plan on attending if you're leading a song.

If you have any questions, feel free to email me at [anya.tche@gmail.com](mailto:anya.tche@gmail.com).

P.S. If you're interested in reading a speech, see [Ozy's post](https://www.lesswrong.com/posts/DoT6G6raWWuSqpdR7/bay-winter-solstice-2024-speech-auditions) with audition instructions for those! Note that speech auditions are due by **November 17**.
# Inferential Game: The Foraging (Ex-)Bandit

The encampment is empty when you awaken. This isn’t necessarily a bad sign, as you weren’t sure you’d wake up at all: there were a *lot *of ways your fellow bandits could have taken your impassioned impromptu speech exhorting them to “be a bit more Robin Hood about the whole thing”, and departing in the dead of night was by no means the worst. Between your continued existence, and the *surprisingly* generous amount of weapons and ammo which were left behind with you, you find yourself feeling rather relieved, and even a little grateful.

These feelings last until you visit the nearest town – hoping to trade bullets for food and a ticket home, so you can forget this sorry chapter of your life – and notice WANTED posters featuring your face on every wall. Your *brief* and *understandably tense* interactions with the locals clarify that  you have been named as your erstwhile gang’s ringleader, that you are emphatically unwelcome in any of the shops in town, that a team of unstoppable mercenaries will be coming for you in a little under two months, and that “I assumed they were dashing desperados when I joined” is *not* typically considered an extenuating factor.

Your only remaining option is to forage for food and head home on foot, before the bounty hunters show up. Fortunately, your campsite was chosen for its closeness to an abundance of easy foraging sites, and you’ve already figured out how to safely and reliably preserve and prepare the ingredients sourced from them. Unfortunately, you were too busy cooking to help your fellow bandits forage[^voral7ki5u], so you have no idea which sites yielded which yields in what quantities; also, the only viable path back has no opportunities to replenish your stockpile en route.

You have sixty days to explore and exploit the sites around the encampment, hunting[^ez79c32jtjq] and gathering enough food to keep yourself alive, and stockpiling enough to reach safety before starving. [Good luck!](https://h-b-p.github.io/Drifting-Bandit/)

Notes
-----

*   This game shouldn’t take more than ~10min to play, runs in-browser, and is not intended to be replayed.
*   Unlike in the classic bandit problem, outcomes can be affected by chronological effects and/or earlier choices. Figuring out where and how this happens is an intended part of the challenge.
*   Data Science skills might be useful here, but will not be necessary.
*   You are warmly encouraged to share and compare completed game records in the comments.
*   Feedback, as always, is greatly appreciated.

[^voral7ki5u]: You’d originally thought they were asking you to be their chief, and by the time you figured out your mistake it was too awkward to back down. 

[^ez79c32jtjq]: You are sufficiently well-armed that nothing you hunt will be able to harm you. Indeed – you consider, feeling your revolver in its place on your belt between your other revolver and your other other revolver – you could justifiably call your current predicament a multi-armed (ex-)bandit problem.
# 2024 NYC Secular Solstice & Megameetup

Secular Solstice is a celebration of hope in darkness. For more than a decade now, people have gathered in New York City to sing about humanity's distant past and the future we hope to build. You are, of course, invited. 

This year, Solstice and the traditional Rationalist Megameetup will both be at the Sheraton Brooklyn New York Hotel, 228 Duffield Street Brooklyn, on the weekend of December 14. We will have sleeping space (Friday, Saturday, and Sunday nights) for those from out of town, as well as meeting space, attendee-organized events, and the ever-popular Unconference. 

[Learn more, register, and get your tickets](https://rationalistmegameetup.com/?utm_source=Rationalist&utm_medium=LesswrongEvent&utm_campaign=2024_Megameet) today!
# Registrations Open for 2024 NYC Secular Solstice & Megameetup

On December 14th, New York City will have a [Secular Solstice](https://www.lesswrong.com/events/ZvWAhkFjqewrEXshg/2024-nyc-secular-solstice-and-megameetup). Solstice is a holiday for people comfortable with uncomfortable truths and who believe in good. Secular Solstices take place in many cities around the world, but for us New York City is the best place for it. The first Solstice started here, amid towers that reach for the sky and glitter like stars in the night. Now a tradition spanning over a decade, rationalists from across North America—and sometimes further afield—will come together to sing about humanity's distant past and the future we hope to build.

From the evening of December 13th to the morning of December 16th, NYC will also be the home of the Rationalist Megameetup. We will have sleeping space (Friday, Saturday, and Sunday nights) for those from out of town, as well as meeting spaces to congregate in. Historically the Megameetup has had calibration games, live podcasting, and the ever-popular lightning talks. There's been scheduled replications of papers, unscheduled lectures on poetry, and more interesting conversations than anyone has counted. 

Solstice and Megameetup will both be at the Sheraton Brooklyn New York Hotel, 228 Duffield Street Brooklyn.

Last year's experiment was whether Megameetup could survive a transition into a conference hotel. The answer was a clear "YES", and we predictably hit capacity on that space like we have every space we've been in since 2019. This year we're back, bigger than before, and with a conference space large enough that Megameetup can host Solstice.

Solstice and Megameetup [registrations](https://rationalistmegameetup.com/register/?utm_source=Rationalist&utm_medium=Lesswrong&utm_campaign=2024_Megameet) are combined, though you can register for only one if you like. Use the[ payment page](https://rationalistmegameetup.com/checkout/?utm_source=Rationalist&utm_medium=Lesswrong&utm_campaign=2024_Megameet) to pick which combination of Solstice and Megameetup events you prefer. We have arranged a hotel block of rooms at a discount, the link for which will be active soon. If we've run out of rooms in the block, you might try booking with the hotel directly. If you do, let them know you're with the New York City Rationalist Megameetup.

We hope to see you there!

-Skyler & The 2024 Megameetup Team