Harley-ml/lesswrong huggingface dataset:
# Fluent dreaming for language models (AI interpretability method)

This is a cross-post for our paper on fluent dreaming for language models. [(arXiv](https://arxiv.org/abs/2402.01702) [link](https://arxiv.org/pdf/2402.01702.pdf)[.)](https://arxiv.org/abs/2402.01702) Dreaming, aka "feature visualization," is a interpretability approach popularized by DeepDream that involves optimizing the input of a neural network to maximize an internal feature like a neuron's activation. We adapt dreaming to language models.

Past dreaming work almost exclusively works with vision models because the inputs are continuous and easily optimized. Language model inputs are discrete and hard to optimize. To solve this issue, we adapted techniques from the adversarial attacks literature (GCG, [Zou et al 2023](https://arxiv.org/abs/2307.15043)). Our algorithm, Evolutionary Prompt Optimization (EPO), optimizes over a Pareto frontier of activation and fluency:

![](https://confirmlabs.org/posts/dreamy_files/figure-html/cell-8-output-1.png)

In the paper, we compare dreaming with max-activating dataset examples, demonstrating that dreaming achieves higher activations and similar perplexities to the training set. Dreaming is especially exciting because some mildly out-of-distribution prompts can reveal details of a circuit. For example, Pythia-12B layer 10, neuron 5 responds very strongly to "f. example", "f.ex." and "i.e" but responds even more strongly to "example f.ie.", a phrase the model has probably never seen in training.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b38bba7c2f32bd5f4ec30dd45f3f7ef44473d81a04eba23c.png) *Figure: Comparing activation and cross-entropy between dreaming outputs and the top 64 max-activating dataset examples from 500 million tokens of the Pile. Lower cross-entropy prompts are more fluent. The black line is schematically separating regions of the plot that are empirically inside and outside the training distribution.*

Like max-activating dataset examples, language model dreams will be hard to interpret in the face of polysemanticity. We would be excited about applying dreaming to more monosemantic feature sets resulting from dictionary learning/sparse autoencoders.

We think algorithms like EPO will also be useful for fluent algorithmic redteaming. We are working on that now!

We have a [companion page here demonstrating the code](https://confirmlabs.org/posts/dreamy.html). We also have [a demo Colab notebook here](https://colab.research.google.com/drive/1B0dM7du91BUkT7tSICXjKL7lrBAEdSa-?usp=sharing).
# How can I efficiently read all the Dath Ilan worldbuilding?

I liked the April Fools Confession post, I liked the AMA, and I liked that long passage somewhere in planecrash about how they reacted to the discovery of the AI problem. But I wasn't really hooked by the glowfic, and only made it 30 pages in.

**Is there a compilation of Dath Ilan worldbuilding links somewhere?** I would like to become confident that I have read all the relevant exerpts from the glowfic and everything else.
# Full Driving Engagement Optional

Yesterday the Secretary of Transportation [wrote](https://twitter.com/SecretaryPete/status/1754559027688685941):

> ALL advanced driver assistance systems available today require the human driver to be in control and fully engaged in the driving task at all times.

While the driver who prompted this was extremely reckless, driving while wearing a 3D headset ([video](https://twitter.com/ayeejuju/status/1754062178284679295)), Buttigieg is wrong here.

[Advanced driver assistance systems](https://en.wikipedia.org/wiki/Advanced_driver-assistance_system) can cover a range of things, from [anti-lock brakes](https://en.wikipedia.org/wiki/Anti-lock_braking_system) to [adaptive cruise control](https://en.wikipedia.org/wiki/Adaptive_cruise_control) and up. For the last several months, however, Americans have been able to buy Mercedes vehicles equipped with the Level 3 [Drive Pilot](https://www.mbusa.com/en/owners/manuals/drive-pilot) which, under very specific circumstances \[1\], do not require the person in the driver seat to be engaged in the task of driving. Instead they can legally (and, according to Mercedes, safely) read, play games, or watch videos, as long as they're prepared to retake control of the vehicle with ten seconds of notice.

Now, this is really a minor caveat, and even if you were in a car with Drive Pilot you couldn't safely wear a headset. But this also seems like a situation where the Secretary of Transportation should be up to date on technological and regulatory advancements.

  
\[1\] On specific highways in CA and NV, in traffic that limits speeds to under 40mph, during the day, in good weather, etc.
# Why I think it's net harmful to do technical safety research at AGI labs

IMO it is harmful on expectation for a technical safety researcher to work at DeepMind, OpenAI or Anthropic.  
  
Four reasons:

1.  **Interactive complexity.** The intractability of catching up – by trying to invent general methods for AI corporations to somehow safely contain model interactions, as other engineers scale models' combinatorial complexity and outside connectivity.  
     
2.  **Safety-capability entanglements**
    1.  *Commercialisation*. Model inspection and alignment techniques can support engineering and productisation of more generally useful automated systems.
    2.  *Infohazards*. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.  
         
3.  **Shifts under competitive pressure**
    1.  DeepMind merged with Google Brain to do commercialisable research,  
        OpenAI set up a company and partnered with Microsoft to release ChatGPT,   
        Anthropic [pitched](https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-to-take-on-openai/#:~:text=%E2%80%9CClaude%2DNext%E2%80%9D%20%E2%80%94%2010%20times%20more%20capable%20than%20today%E2%80%99s%20most%20powerful%20AI%2C%20but%20that%20this%20will%20require%20a%20billion%20dollars%20in%20spending%20over%20the%20next%2018%20months.) to investors they'd build a model 10 times more capable.
    2.  If you are an employee at one of these corporations, higher-ups can instruct you to do R&D you never signed up to do.[^otsrraiw02j] You can abide, or get fired.
    3.  Working long hours surrounded by others paid like you are, by a for-profit corp, is bad for maintaining bearings and your epistemics on safety.[^yofil7yaxwm]  
         
4.  **Safety-washing**. Looking serious about 'safety' helps labs to recruit idealistic capability researchers, lobby politicians, and market to consumers.
    1.  'let's build AI to superalign AI'
    2.  'look, pretty visualisations of what's going on inside AI'  
         

This is my view. I would want people to engage with the different arguments, and think for themselves what ensures that future AI systems are actually safe.

[^otsrraiw02j]: I heard via via that Google managers are forcing DeepMind safety researchers to shift some of their hours to developing Gemini for product-ready launch.I cannot confirm whether that's correct.  

[^yofil7yaxwm]: For example, I was in contact with a safety researcher at an AGI lab who kindly offered to read my comprehensive outline on the AGI control problem, to consider whether to share with colleagues. They also said they're low energy. They suggested I'd remind them later, and I did, but they never got back to me. They're simply too busy it seems.
# How to deal with the sense of demotivation that comes from thinking about determinism?

What I know about science and philosophy suggests that **determinism** is probably true - i.e., everything that will happen throughout the lifetime of the universe was predetermined at the beginning of the universe.

When I think about this I get a sense of demotivation and pointlessness. If I know (or assume) that everything I will do is predetermined, that undermines and weakens my motivation to do anything. Why bother with effort and hardship if, at the end of the day, I will always do the one and only thing I was predetermined to do anyway?

When I need to motivate myself to do anything that takes effort, I need to either not think about determinism at all or tell myself that *determinism might be false*.

Does anyone else feel this sense of demotivation? If so, how do you deal with it?

When I have asked this question elsewhere in the past I have mostly gotten answers that were unhelpful or even downright mocking. This leads me to suspect that the sense of demotivation I get is not a human universal but something that comes naturally to *some* people and is completely alien to others. Therefore, if the problem seems easy to you, I would ask you to please think really hard about whether you are actually helping before you post. 

And please, for the figurative love of God, no joke replies.

Note that I did not use the term "free will" in the above. One can assume a compatibilist stance and argue that our choices are still "free" even though they are determined. As far as I can tell, that is a word game which does not solve the problem. At least, I have met no explanation of determinism that assuaged my feeling of demotivation.

I previously asked a variant of this question in r/AskPhilosophy on Reddit: [https://www.reddit.com/r/askphilosophy/comments/rvvyc4/what\_to\_do\_about\_the\_fatalist\_argument\_that\_if/](https://www.reddit.com/r/askphilosophy/comments/rvvyc4/what_to_do_about_the_fatalist_argument_that_if/)   
 

Thanks in advance!
# What's this 3rd secret directive of evolution called? (survive & spread & ___)

Is a species' abundance determined solely by its ability to survive and spread? Almost but not quite (I claim). There is a third requirement: it must come to exist in the first place! Furthermore, spontaneous formation alone can suffice with very poor surviving & spreading.

Example 1: verrry early life in the soup was probably better at randomly being created than it was at surviving or spreading.

Example 2: photons aren't much good at surviving or spreading but hey I can see in here

Example 3: some ideas are popular because they get rediscovered all the time (rather than spreading from the original source) — eg good ways to wipe your butt. Oh and crabs!

I think these three directives are necessary and sufficient. In other words, abundance is determined by how long you survive, how fast you spread, how often you pop up out of nowhere, and nothing else. (Am I wrong?)

**What would (or did) someone call this third thing? Would be nice if it fits "survive & spread & ___". 'Convergent evolution' isn't the right fit... 'reachability'? 'naturalness'? 'inevitability'?**

(While we're at it, does anyone have a term for 'natural selection' when you're not just talking about plants & animals etc? Don't really like dragging that term up to such a broad level. A "general theory of abundance" or something¿)
# More Hyphenation

> “MAN EATING PIRANHA MISTAKENLY SOLD AS PET FISH” — example news headline from Steven Pinker’s *The Sense of Style*

The rule is that you use hyphens for compound modifiers like the ones in natural-language processing, high-impact opportunities, cost-effectiveness measures, high-status employers, and so on. Don’t break up compound proper nouns (“New York-based company”) and don’t use them after adverbs ending in -ly but do use them after other adverbs (“stern-looking boss”). You can use suspended hyphens when talking about “latex- and phthalate-free gloves.”

But hyphens are under attack. The *Chicago Manual of Style* “prefers a spare hyphenation style.” The *AP Stylebook* says that [“the fewer hyphens the better.”](https://twitter.com/APStylebook/status/1167125311995203584) In older texts you see a lot more hyphenation than you do today.

Part of this is because of a good trend of combining compound nouns, turning e-mail and fire-fly into email and firefly. But part of it involves replacing hyphens with spaces, turning high-school seniors and ice-cream cones into high school seniors and ice cream cones. Some people think hyphens just look bad.

But hyphens are excellent because they improve the readability of text—the speed at which it can be understood, even at a less-than-perceptible level. In fact, it would probably be an improvement to language if it became acceptable and normal to hyphenate compound nouns simply to make the noun phrase faster to read. But first I hope we can return to making references to chocolate-chip cookies.

Skimming the curated posts that are on LessWrong right now, as a random sample:

*   A Shutdown Problem Proposal → A Shutdown-Problem Proposal
*   hopefully-corrigible agent → hopefully corrigible agent
*   large scale X → large-scale X

A good example of hyphen use: “to make any child-agents it creates responsive-but-not-manipulative to the shutdown button, recursively.”
# How do health systems work in adequate worlds?

I'm confused about Health Systems in general, the same way I was confused about the Education System until I read "The case against education". I don't have a good model of what are the different agents interacting in this system, nor the differences among different countries. Is there any reading you would recommend to be a bit less confused about this?
# How do high-trust societies form?

First, it's clear to me they're not just myths bolstered by propaganda, and they really *are* qualitatively different from their low-trust counterparts, which I understand, and consider to be "natural," requiring no explanation.

I think it's a different concept than the WEIRD (Western, Educated, Industrialized, Rich, Democratic) thing, though of course, there's a lot of overlap.

A related question: how do they *unravel,* and can this be arrested*?*
# Biden-Harris Administration Announces First-Ever Consortium Dedicated to AI Safety

"Today, U.S. Secretary of Commerce Gina Raimondo announced the creation of the [U.S. AI Safety Institute](https://www.nist.gov/artificial-intelligence/artificial-intelligence-safety-institute) Consortium (AISIC), which will unite AI creators and users, academics, government and industry researchers, and civil society organizations in support of the development and deployment of safe and trustworthy artificial intelligence (AI)"

It's not clear to me whether this is *also* the first announcement of the "US AI Safety Institute" as well as the Consortium as a possible separate thing, but either way, this seems like great news.
# Constituency-sized AI congress?

I just had the idea for a constituency-sized AI congress. Each member of the constituency would have their personal debate agent trained on their preferences and values. The agents would debate tirelessly at superhuman speed to develop proposals which represented the best available win-win compromises given the issues on the table. 

The final proposals after a set amount of debate would be presented to the constituency and voted on.

I haven't researched this yet or thought about it for long. I'd love for your feedback on the idea and links to related work.
# Close the Gates to an Inhuman Future: How and why we should choose to not develop superhuman general-purpose artificial intelligence

This is a revised version of a paper I put out late last year (including a new table about capabilities lacking in current AI and how we might get them.)

Interested in reactions and potential improvements, as I'm contemplating various possibilities for either expansion or distillation/repackaging of this basic picture, for a variety of audiences.
# What's the theory of impact for activation vectors?

[Activation vectors](https://www.lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector) are really, really cool, but what is the theory of impact for this work?

*   Is the hope that activation vectors will allow us to actually gain perfect control over a model to get it to do exactly what we want it to do?
*   Is the hope that a new technique that builds upon activation vectors lets us do that instead?
*   Is the hope that this technique allows us to marginally decrease the risks of powerful models in a Hail Mary attempt? Or perhaps to buy us more time to solve the problem?
*   Is the hope just that learning more about how neural networks work will allow us to theorize better about how to control them?
# How do you actually obtain and report a likelihood function for scientific research?

I just read the article [https://arbital.com/p/likelihoods\_not\_pvalues/,](https://arbital.com/p/likelihoods_not_pvalues/,) but this left me with one big question: what is the actual sequence of actions a researcher would take to report a likelihood function based on their research? For background, I'm an undergraduate student who has participated in multiple undergraduate research positions, and who has so far been uncertain about how to compute and use likelihood functions based on the data (mostly econometric data) involved in this research.

In particular, I want to find out:

1.  How do you compute a likelihood function in Stata, R, Python, etc. based on data you have, for a given hypothesis class
2.  How do you obtain a data file for this likelihood function that can be distributed in a way that other researchers can download and use in e.g. a meta-analysis
3.  What kinds of graphs are good to visually represent these likelihood functions, particularly if the hypothesis class has multiple parameters, and
4.  What kinds of summary statistics are good to numerically represent these likelihood functions in a human-readable way
5.  How has this whole thing historically been done in fields like machine learning that have historically been more amenable to reporting likelihood functions

If you have good answers for any of these questions, I'd really like to know.
# What are the deciding factors of human cognitive endurance?

Cal Newport says something to the effect that the median capacity for high-intensity focus is about four hours per day in *Deep Work.* I think this can bounce around over time for individuals – especially regarding psychological dispositions like bipolar or autism, which proffer an immense capacity for hyper-fixation at the cost of occasional catatonia. Aside from fluctuation, there's remarkable variance between people.

On the high-end, some people can maintain their peak-intensity on a regular basis for ten hours (e.g. John Carmack) without burning out long-term or experiencing short-term crashes. On the low-end, despite a healthy life-style, some people struggle to stay awake during the day (search "you definitely have a thyroid problem" on r/productivity).

What are the deciding factors of this variance, that stake out interventions or practices people can implement to dramatically improve this ability? 

* * *

\[**edit/add**: drugs are another frame of cognitive endurance – modafinil can, within hours, bring a person to a level of concentration and energy further than a healthy diet and exercise could perhaps ever take that person. Is there any robust method to achieve modafinil-enhanced performance as a baseline – and without chemical intervention?

I've noticed that general cognitive endurance matters to performance more than personal interest, *and* that I have high cognitive endurance for some things but at times zero for others, which makes it so that I can spend the entire day doing something that looks like intense work on the outside, but is internally experienced as "just another minute"-style procrastination. This is frustrating, and I've had to drop many classes for fatigue reasons.

It's also interesting that there isn't a specific guide to maintaining high-levels of general cognitive endurance, given its economic value – though high-demand companies and startups already select from a pool of high-endurance people.\]
# What are the known difficulties with this alignment approach?

Assume you have a world-model that is nicely factored into spatially localized variables that contain interesting-to-you concepts. (Yes, that's a big assumption, but are there any known difficulties with the proposal if we grant this assumption?)

Pick some Markov blanket (which contains some actuators) as the bounds for your AI intervention.

Represent your goals as a causal graph (or computer program, or whatever) that fits within these bounds. For instance if you want a fusion power plant, represent it as something that takes in water and produces helium and electricity.

Perform a Pearlian counterfactual surgery where you cut out the variables within the Markov blanket and replace them with a program representing your high-level goal, and then optimize the action variables to match the behavior of the counterfactual graph.
# Where is the Town Square?

I am seeking crowdsourced wisdom.

Suppose I[^sa06s0ni0ld] want to influence public opinion on a complicated, nuanced topic.[^l4imxgu66c]

And further suppose that one of the ways I want to do that is by participating actively in public discourse: posting short-form content, reacting to other people's short-form content, signal boosting good takes, thoughtfully rebutting bad takes, etc.

Suppose that the people I most want to reach are those who make and influence policy in places like the US, EU, UK, and China; the general voting public (because they vote for legislators); and the intelligentsia in fields related to my topic (because some of them advise policymakers).

On what platform(s)/in what outlet(s) should I be doing this in 2024? "And, since I can't do everything: what popular platforms **shouldn't** I prioritize?

And more specifically, does Twitter/X still matter, and how much? I am aware that many people have moved to mastodon or bluesky or whatever. Is there critical mass anywhere?

[^sa06s0ni0ld]: who happen to be the Communications Manager at MIRI 

[^l4imxgu66c]: The topic is AI x-risk.
# FTX expects to return all customer money; clawbacks may go away

[New York Times (unpaywalled)](https://www.nytimes.com/2024/02/03/business/dealbook/what-happens-to-ftx-clawback-cases-if-the-company-repays-its-creditors.html?unlocked_article_code=1.VU0.QBLc.h36dKm_EgU8L&smid=url-share):

> When the cryptocurrency exchange FTX declared bankruptcy about 15 months ago, it seemed few customers would recover much money or crypto from the platform. As John Ray III, who took over as chief executive during the bankruptcy, put it, “At the end of the day, we’re not going to be able to recover all the losses here.” He was countering Sam Bankman-Fried’s repeated claims that he could get every customer their money back.
> 
> Well, it turns out, FTX lawyers told a bankruptcy judge this week that they expected to pay creditors in full, though they said it was not a guarantee and had not yet revealed their strategy.
> 
> The surprise turn of events is raising serious questions about what happens next. Among them: What does this mean for the lawsuits FTX has filed in an attempt to claw back billions in assets that the company says it’s owed?
> 
> Will the possibility that customers could be made whole be raised at Bankman-Fried’s sentencing? Will potential relief for customers help his appeal?
> 
> \[...\]
> 
> **Some of the clawback cases involve allegations of fraud, but not all do.** Before fraud claims are argued, there is typically a legal fight over whether a company was insolvent at the time of the investment or that the investment led to insolvency. If every FTX creditor stands to get 100 cents on the dollar, the clawback cases that don’t involve fraud wouldn’t serve much of a financial purpose and may be more difficult to argue, some lawyers say. “In theory, clawbacks may go away there,” said Eric Monzo, a partner at Morris James who focuses on bankruptcy claims.

[Court proceedings](https://www.documentcloud.org/documents/24408442-ftx-trading-ltd-013124-2):

> we can now cautiously predict some measure of success. Based on our results to date and current projections we anticipate filing a disclosure statement in February describing how customers and general unsecured creditors, customers and general unsecured creditors with allowed claims, will eventually be paid in full. I would like the Court and stakeholders to understand this not as a guarantee, but as an objective. There is still a great amount of work and risk between us and that result, but we believe the objective is within reach and we have a strategy to achieve it.
# What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

It started with this video of Hinton taking a jab at Marcus: [https://twitter.com/tsarnick/status/1754439023551213845](https://twitter.com/tsarnick/status/1754439023551213845)

And here is Marcu's answer:

[https://garymarcus.substack.com/p/deconstructing-geoffrey-hintons-weakest](https://garymarcus.substack.com/p/deconstructing-geoffrey-hintons-weakest)

As far as I understand, Gary Marcus argues that LLMs memorize some of their training data, while Hinton argues that no such thing takes place, it's all just patterns of language.

I found these two papers on LLM memorization:

[https://arxiv.org/abs/2202.07646](https://arxiv.org/abs/2202.07646) \- **Quantifying Memorization Across Neural Language Models**

[https://browse.arxiv.org/abs/2311.17035](https://browse.arxiv.org/abs/2311.17035) \- **Scalable Extraction of Training Data from (Production) Language Models**

Am I missing something here? Are these two positions compatible or does one need to be wrong for the other one to be correct? What is the crux between them and what experiment could be devised to test it?
# Disrupting malicious uses of AI by state-affiliated threat actors

OpenAI just released a public announcement detailing how they caught and disrupted several cases of ongoing misuse of their models by state-affiliated threat actors, including some known to be affiliated with North Korea, Iran, China, and Russia.

This is notable because it provides very tangible evidence of many kinds of misuse risk that many people in AI Safety had flagged in the past (like the [use of LLMs for aiding in the development of spear-fishing campaigns](https://arxiv.org/abs/2305.06972)), and it associates them with malicious state-affiliated groups.

The specific findings:

> Based on collaboration and information sharing with Microsoft, we disrupted five state-affiliated malicious actors: two China-affiliated threat actors known as Charcoal Typhoon and Salmon Typhoon; the Iran-affiliated threat actor known as Crimson Sandstorm; the North Korea-affiliated actor known as Emerald Sleet; and the Russia-affiliated actor known as Forest Blizzard. The identified OpenAI accounts associated with these actors were terminated.
> 
> These actors generally sought to use OpenAI services for querying open-source information, translating, finding coding errors, and running basic coding tasks. 
> 
> Specifically: 
> 
> *   Charcoal Typhoon used our services to research various companies and cybersecurity tools, debug code and generate scripts, and create content likely for use in phishing campaigns.
> *   Salmon Typhoon used our services to translate technical papers, retrieve publicly available information on multiple intelligence agencies and regional threat actors, assist with coding, and research common ways processes could be hidden on a system.
> *   Crimson Sandstorm used our services for scripting support related to app and web development, generating content likely for spear-phishing campaigns, and researching common ways malware could evade detection.
> *   Emerald Sleet used our services to identify experts and organizations focused on defense issues in the Asia-Pacific region, understand publicly available vulnerabilities, help with basic scripting tasks, and draft content that could be used in phishing campaigns.
> *   Forest Blizzard used our services primarily for open-source research into satellite communication protocols and radar imaging technology, as well as for support with scripting tasks.
# 11 diceware words is enough

A tweet-thread (X thread?) by [@benwr](https://www.lesswrong.com/users/benwr?mention=user) about how many words you need if you want to use the [diceware system](https://www.eff.org/dice) to create a password that will be safe for 20 years against 1 year of the whole world's password-cracking capacity. He estimates that 11 diceware words suffices. Content replicated below, with minor editing:

> If you care about password strength: you want to measure strength in terms of the [min-entropy](https://www.benwr.net/2022/01/16/password_entropy.html) of the randomized method you use to choose. But how much min-entropy should a password have?
> 
> Well, I use a password manager to generate and store ~all of my passwords. I don't care how hard they are to memorize, so I generate crazy-long ones. But there's an important exception: The password for my password manager! How strong does that one need to be?
> 
> Important questions: "how long do I want to be safe for?" and "safe against what kind of attacker?". But, if you want to ignore this question and be safe anyway, one option is "I want to be safe for 20 years, against one year of the entire world's cracking capacity."
> 
> Let's make some crazy-conservative assumptions:
> 
> 1.  A cracking attempt can be done with a single floating-point op
> 2.  N quantum gates can do the equivalent of N^2 floating-point ops worth of password cracking attempts, using Grover's algorithm
> 3.  compute doubles every year
> 
> Globally < ~10^24 = 2^80 classical ops/s. Quantum gates is harder to estimate, but I think currently less than 10^8 = 2^27 gates/s. If both double yearly, classical will dominate for about 15 years. But since we want 20 years of security, we care more about quantum.
> 
> 2^27 gates/s doubled for 20 years gives 2^47 gates/s. pi seconds is a nanocentury: 3.15e7 seconds in a year, so\[, since 3.15e7 = 30 * 10^6 = 2^5 * 2^20 = 2^25\], 2^25 * 2^47 = 2^72 gates/year. Double that (quantum, remember?) to get 144 bits of cracking capacity. ~11 diceware words. Not impossible to remember!
> 
> Estimates of global classical compute:
> 
> *   [How much computing capacity exists in GPUs and TPUs in Q1 2023?](https://wiki.aiimpacts.org/ai_timelines/hardware_and_ai_timelines/computing_capacity_of_all_gpus_and_tpus), from the [AI Impacts Wiki](https://wiki.aiimpacts.org/start)
> *   [Global computing capacity](https://aiimpacts.org/global-computing-capacity/), from [AI Impacts](https://aiimpacts.org/#gsc.tab=0)
> 
> Estimate of global quantum compute is based on [this table](https://en.wikipedia.org/wiki/List_of_quantum_processors)
> 
> plus [this estimate](https://quantumcomputing.stackexchange.com/posts/2404/revisions) of gates/s for a single machine
> 
> Compute doubling yearly is a wild and crazy number I made up to be way too fast
> 
> [Diceware](https://theworld.com/~reinhold/diceware.html)
> 
> Addendum: Every additional diceware word you add should increase the timeframe by about 6 years
# "Open Source AI" isn't Open Source

Open source software has long differentiated between “free as in speech” (libre) and “free as in beer” (gratis). In the first case, libre software has a license that allows the user freedom to view the source and modify it, understand it, and remix it. In the second case, gratis software does not need to be paid for, but the user doesn’t necessarily have access to the pieces, can’t make new versions, and cannot remix or change it.  
  
...  
  
If Open Source AI is neither gratis or libre, then those calling free model weights “Open Source,” should figure out what free means to them. Perhaps it’s “free as in oxygen” (dangerous due to reactions it can cause), or “free as in birds” (wild, without any person responsible).

I’m not necessarily opposed to judicious release of model weights, though as with any technology, designers and developers should consider the impact of their work before making or releasing it, as LeCun has recently agreed. But calling this new competitive strategy by Facebook “Open Source” without insisting on the actual features of open source is an insult to the name.
# Fatebook for Chrome: Make and embed forecasts anywhere on the web

[Fatebook](https://forum.effectivealtruism.org/posts/DWFRBzK3rAH3HFDZr/fatebook-the-fastest-way-to-make-and-track-predictions) is the fastest way to track your predictions. Now we've made a Chrome extension that makes it even faster.

With [**Fatebook for Chrome**](https://fatebook.io/extension), you can now create and embed forecasts inside Google Docs.

![https://stratus.campaign-image.eu/images/136129000001239684_2_1706807358166_(9).png](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/Rbh9xHCyej4sL9jJx/qi5ijdc8psirwkshnlqv)

Or anywhere else on the web! Inside your to-do list...

  
![https://stratus.campaign-image.eu/images/136129000001239684_2_1706807358166_(9).png](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/Rbh9xHCyej4sL9jJx/k7bgomiqhabmase9dcrl)

Or even inside Google Meet!

  
![https://stratus.campaign-image.eu/images/136129000001239684_2_1706807358166_(9).png](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/Rbh9xHCyej4sL9jJx/s373bynmtfvuvfn0zows)

To instantly create a forecast on any webpage, just press Ctrl-Shift-F, type your prediction, and hit enter:

  
![https://stratus.campaign-image.eu/images/136129000001239684_2_1706807358166_(9).png](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/Rbh9xHCyej4sL9jJx/ds26dewb31fckjgxkppy)

Imagine you're writing a Google Doc – a report on the rate of AI progress. You want to write down a prediction: "*The most capable LLM in 2026 will be made by OpenAI (80%)"*

With Fatebook for Chrome, you can press **Ctrl-Shift-F**, write your prediction, and embed it right into your doc. Your forecast is recorded in Fatebook, so you won't lose track of it: you'll get a reminder to resolve it in 2026, and your accuracy counts towards your track record.

And for your colleagues reading your report, they'll see your prediction and can add their own, letting you harness the wisdom of the crowd, understand disagreements, and update your beliefs over time.

This is the next step on our mission to make Fatebook the fastest way to make and track predictions, and to make it super low friction to bring forecasting right into your team's workflows.

We're excited to hear how you use it – particularly if you're using it with your team, as we designed this tool with EA orgs and other high-impact teams in mind. Special thanks to [@Jarred Filmer](https://www.lesswrong.com/users/jarred-filmer?mention=user) for his awesome engineering work to bring Fatebook into any site on the web!

You can [**add Fatebook for Chrome to your browser**](https://fatebook.io/extension) (whether you use Chrome, Edge, or Arc), and you can see the extension in action here:  [**fatebook.io/extension**](https://fatebook.io/extension)
# Physics-based early warning signal shows that AMOC is on tipping course

**Abstract**
------------

One of the most prominent climate tipping elements is the Atlantic meridional overturning circulation (AMOC), which can potentially collapse because of the input of fresh water in the North Atlantic. Although AMOC collapses have been induced in complex global climate models by strong freshwater forcing, the processes of an AMOC tipping event have so far not been investigated. Here, we show results of the first tipping event in the Community Earth System Model, including the large climate impacts of the collapse. Using these results, we develop a physics-based and observable early warning signal of AMOC tipping: the minimum of the AMOC-induced freshwater transport at the southern boundary of the Atlantic. Reanalysis products indicate that the present-day AMOC is on route to tipping. The early warning signal is a useful alternative to classical statistical ones, which, when applied to our simulated tipping event, turn out to be sensitive to the analyzed time interval before tipping.
# Weighing reputational and moral consequences of leaving Russia or staying

I've been thinking a lot about whether I should consider emigrating to another country (currently I'm in Russia). I'm not involved in politics or the military action, but I can't help but feel implicated by my government's actions as a citizen here. Specifically, the increasing authoritarianism and human rights issues worry me. On the other hand, I have family and (very few) friends in Russia, plus clearer studying opportunities, and I'm not sure if I want to leave that behind; also, Russia is my home - I'm unsure whether this might be important.

I'm a software engineer in my 20s living in Moscow, with self-employment and quite stable income. I'm interested in building global startups, so I'm worried about the reputation of my future company(ies) in the context of remaining in Russia for so long. Meanwhile, I'd prefer to continue cooperating with my (small) Russian social circle. But I'm unsure they have my better interests in mind (so I'd also like advice on how to determine that, if possible).

I'm looking for perspectives from this community on the moral and other considerations and responsibilities I have as an individual. Mainly I worry about global reputation of myself and my (future) companies. Should I continue my life here as usual? Should I keep contacts and cooperation with acquaintances/friends (some of whom might be potential business partners) who remain? (Btw, I somewhat worry that I could absorb some of their differing beliefs which could make me less effective.) Is there anything constructive I can do if I stay? Or is leaving the country and minimizing contacts the only ethical choice?

Curious to hear thoughts. I'm especially interested in advice on how to weigh all this from those with more experience in similar situations.

I don't have strong opinions yet, so please share a variety of perspectives. Mostly looking for help in thinking this through in a thoughtful way during a confusing time.
# Abs-E (or, speak only in the positive)

"Positive" and "negative", as used here, refer to truth-values and logical structure, rather than emotional valence.

On almost any topic, there are many potential false statements, and comparatively very few[^soy8gnezvk] true statements. Thus a [simple true statement (like "the sky is blue") is usually much more informative](https://www.lsusr.com/blog/speak-in-the-positive.html) than the negation of a similar false statement (like "the sky is not green").

When reading or hearing a negation used in language, you must first process the positive form it contains to understand the entire statement. For example, to understand "the sky is not green", you must first understand "the sky is green", then negate it. Usually, this happens quickly and subconsciously, but it can harmfully slow down or weaken understanding by making you first consider a false idea.

For these reasons, Absolute-English (henceforth Abs-E), where one only speaks in the positive, should be clearer and more honest than current negation-permitting English. I call it Abs-E by analogy to the absolute-value operation in maths, which replaces both positive and negative with only positive. The simple form of Abs-E forbids the words "no", "not", and "-n't" (as in "won't" or "isn't"). The strict form of Abs-E, which may be more effective, forbids all of the following:

*   "no"
*   "not"
*   "cannot"
*   "-n't"
*   "im-", "in-", "non-", "un-", and "-less" where negating adjectives, as in "immortal", "indigestible", "nonsensical", "unfit", and "wireless". Some other adjectives, like "imminent" and "informative", are positive root words on their own.
*   empty quantifiers/negative correlative (as "nothing" or "nowhere")

The proposal is loosely inspired by the existing [E-Prime, which forbids "to be" and all its inflections](https://en.wikipedia.org/wiki/E-Prime) for its own, different reasons. You could combine the restrictions to make Abs-E-Prime, replacing "the sky is not green" with "the sky appears blue", but I only care about positivity.

Under Abs-E, binary questions ("yes"-or-"no") are less obvious to answer. If your answer would ordinarily be "no", you must instead reply as if the question was open-ended. For example, your reply to "will you be here tomorrow?" may be "yes", or "I will be in the office tomorrow", or "I will stay home tomorrow". This forces you to speak with more information. On average, that makes conversation more fun and lying harder. If your interlocutor knows about Abs-E, they can make questions and answers play out more naturally by replacing all binary questions with open-ended questions, such as "where will you be tomorrow?" in place of "will you be here tomorrow?".

I am serious about following strict Abs-E, at least for some trial time to see its effects. If, in my speech or writing, you find me using a negative form, you are welcome to call it out for me to correct.

[^soy8gnezvk]: Pedantically, there are an equal number of true and false statements (both infinite). Given any one maximum length of statements, there are only finitely many true statements and many more (still finite) false statements.
# Job Listing: Managing Editor / Writer

MIRI is hiring a managing editor and one or more writers. Here's the [job listing](https://jobs.ashbyhq.com/miri/e07416be-e761-405c-b46d-e017324108da). We're accepting applications until March 13th.

I am the Communications Manager at MIRI and I'm the hiring manager for these positions. I'm happy to answer questions.
# Embed your second brain in your first brain

Any explicit knowledge is either part of a general, long-term understanding of the world (global knowledge), or peculiar to your own life and local environment (local knowledge). These mainly fill semantic and episodic memory, respectively. Declarative memory can hold either.

Local facts tend to be isolated from each other, and can go obsolete quickly. Natural memory with static linear notes suffices to keep track of local knowledge. Being able to quickly search the notes might help.

Global facts tend to connect to each other, and remain true and relevant for years or even lifetimes. Some propose using Zettelkasten-like external "second brains" to record, reflect on, and connect global facts, such as in [Obsidian](https://obsidian.md/), [Roam](https://roambrain.com/), or [TiddlyWiki](https://tiddlywiki.com/).

You can use a spaced repetition system (SRS, such as [Anki](https://apps.ankiweb.net/) or [Memoire](https://dkl9.net/memoire/)) to efficiently memorise large amounts of explicit knowledge.

1.  Express each fact in a precise, atomic, recorded form.
2.  Load the facts into the SRS.
3.  Review as directed.

Once you memorise facts, you can reflect on them in your brain, perhaps assisted with disposable writing. Spaced repetition also brings an incentive to connect facts. You can recall a fact more easily by rederiving it from others you already know, and spaced repetition rewards you for easily recalling things.

Try as you might to shrink the margin with better technology, recalling knowledge from within is necessarily faster and more intuitive than accessing a tool. When spaced repetition fails (as it should, up to 10% of the time), you can gracefully degrade by searching your SRS' deck of facts.

If you lose your second brain (your files get corrupted, a cloud service shuts down, etc), you forget its content, except for the bits you accidentally remember by seeing many times. If you lose your SRS, you still remember over 90% of your material, as guaranteed by the algorithm, and the obsolete parts gradually decay. A second brain is more robust to physical or chemical damage to your first brain. But if your first brain is damaged as such, you probably have higher priorities than any particular topic of global knowledge you explicitly studied.

I write for only these reasons:

*   to help me think
*   to communicate and teach (as here)
*   to distill knowledge to put in my SRS
*   to record local facts for possible future reference

Linear, isolated documents suffice for all those purposes. Once you can memorise well, a second brain becomes redundant tedium.
# Shaming with and without naming

Suppose someone wrongs you and you want to emphatically mar their reputation, but only insofar as doing so is conducive to the best utilitarian outcomes. I was thinking about this one time and it occurred to me that there are at least two fairly different routes to positive utilitarian outcomes from publicly shaming people for apparent wrongdoings*:

A) People fear such shaming and avoid activities that may bring it about (possibly including the original perpetrator)

B) People internalize your values and actually agree more that the sin is bad, and then do it less

These things are fairly different, and don’t necessarily come together. I can think of shaming efforts that seem to inspire substantial fear of social retribution in many people (A) while often reducing sympathy for the object-level moral claims (B).

It seems like on a basic strategical level (ignoring the politeness of trying to change others’ values) you would much prefer have 2 than 1, because it is longer lasting, and doesn’t involve you threatening conflict with other people for the duration.

It seems to me that whether you name the person in your shaming makes a big difference to which of these you hit. If I say “Sarah Smith did \[—\]”, then Sarah is perhaps punished, and people in general fear being punished like Sarah (A). If I say “Today somebody did \[—\]”, then Sarah can’t get any social punishment, so nobody need fear that much (except for private shame), but you still get B—people having the sense that people think \[—\] is bad, and thus also having the sense that it is bad. Clearly not naming Sarah makes it harder for A) to happen, but I also have the sense—much less clearly—that by naming Sarah you actually get less of B).

This might be too weak a sense to warrant speculation, but in case not—why would this be? Is it because you are allowed to choose without being threatened, and with your freedom, you want to choose the socially sanctioned one? Whereas if someone is named you might be resentful and defensive, which is antithetical with going along with the norm that has been bid for? Is it that if you say Sarah did the thing, you have set up two concrete sides, you and Sarah, and observers might be inclined to join Sarah’s side instead of yours? (Or might already be on Sarah’s side in all manner of you-Sarah distinctions?)

Is it even true that not naming gets you more of B?

—

*NB: I haven’t decided if it’s almost ever appropriate to try to cause other people to feel shame, but it remains true that under certain circumstances fantasizing about it is an apparently natural response.
# Parasocial relationship logic

If:

1.  You become like the five people you spend the most time with (or something remotely like that)
    
2.  The people who are most extremal in good ways tend to be highly successful
    

Should you try to have 2-3 of your five relationships be parasocial ones with people too successful to be your friend individually?