Harley-ml/lesswrong huggingface dataset:
# Switching to a 4GB SD

When I initially switched the computer portion of my [rhythm stage setup](https://www.jefftk.com/news/jammer) over to a Raspberry PI I went with a 32GB SD card because [that's recommended](https://www.raspberrypi.com/documentation/computers/getting-started.html). But I don't need very much space for what I'm doing and a bigger SD card makes copies slow: the cheap cards I've been using take me [over an hour](https://www.jefftk.com/p/duplicating-rasberry-pi-images) to write with my Mac's card reader.

I recently made some changes to my setup, [removing support](https://www.jefftk.com/p/keyboard-gremlins) for bass whistle on the PI (now that I have an [arduino-based version](https://www.jefftk.com/p/embedded-whistle-synth)) and adding support for [more foot-bass options](https://www.jefftk.com/p/simultaneous-footbass-and-footdrums-ii). This normally means round of re-imaging all of my SD cards: I have a bunch of them because SD card corruption is probably the most common reason my system breaks \[1\], and it's easy to have extra cards. I'm pretty sick of 82-minute waits, however, so I decided to go down to four-GB SD cards.

I imaged a card with [Raspberry Pi OS Lite (64-bit)](https://www.raspberrypi.com/software/operating-systems/), started setting it up, and immediately ran out of space. Turns out `apt-get update` and `apt-get upgrade` can use a lot of space. After `apt-get purge` and `apt-get clean`, however, I had about 300MB to spare. It would probably be possible to get a lite distribution to go a lot smaller than this, since the only actually large thing I need is the sound font, [fluid-soundfont-gm](https://packages.debian.org/sid/fluid-soundfont-gm) at 145MB, but this seems to work fine.

Regardless, it works great, and I can now image a card in 15min:

$ time sudo dd if=~/Desktop/rpi-4gb.dmg of=/dev/disk4
7782400+0 records in
7782400+0 records out
3984588800 bytes transferred in 877.723067 secs (4539688 bytes/sec)

real       14m41.556s
user       0m3.299s
sys        0m28.741s

[![](https://www.jefftk.com/rpi-jammer-in-case.jpg)](https://www.jefftk.com/rpi-jammer-in-case-big.jpg)

  
\[1\] This used to happen a lot, but now mostly doesn't. The hardware hasn't changed, so I think the reason it has stopped happening is that my Raspberry PI is no longer responsible for the bass whistle. My MIDI router doesn't write to the file system, and doesn't read beyond initialization, while the bass whistle version used the file system to persist settings across reboots.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0zvQYVuPpbeQm2KP618wJwZQjRJsCTCLZZEZNox54ZefYxw7VgXxuuQAA5FhirAZfl), [mastodon](https://mastodon.mit.edu/@jefftk/113186561472886494)*
# Bounty for Evidence on Some of Palisade Research's Beliefs

*(Cross-posted from the Bountied Rationality Facebook group)*

EDIT: Bounty Expired

*Thanks everyone for thoughts so far! I do want to emphasize that we're actually highly interested in collecting even the most "obvious" evidence in favor of or against these ideas. In fact, in many ways we're more interested in the obvious evidence than in reframes or conceptual problems in the ideas here;  of course we want to be updating our beliefs, but we also want to get a better understanding of the existing state of concrete evidence on these questions. This is partly because we consider it part of our mission to expand the amount and quality of relevant evidence on these beliefs, and are trying to ensure that we're aware of existing work.*

[Here is a Google Doc](https://docs.google.com/document/d/1B7jSmdm891MQa33_9O-2NrtfhiPopOQSFYCAtXHBdXM/edit) that lists 21 important beliefs that Palisade Research has about AI. For each belief, we're looking for the strongest evidence that exists in favor of that idea, and the strongest evidence that exists against it. We'll award at least $20 for the best evidence in favor, and at least $20 for the best evidence against each idea. We'll use our discretion for what we consider the "best" evidence, but the kind of thing we're looking for includes empirical research or convincing arguments. Empirical research, or arguments clearly backed by empirical observations, will be preferred over pure arguments.  
  
To submit a piece of evidence, you can either comment here, making it clear which specific idea(s) you're giving evidence for, or you can add a comment to the linked document. A piece of evidence should include a link, should be clearly associated with a specific idea, and should include a short sentence about how the evidence applies to that idea.  
  
For example, you might write a comment on "**a strategic AI system will aim to appear convincingly aligned with human goals, or incapable of harming humans, whether it really is or not."**, that includes a link to a paper on AI Sandbagging (e.g. "AI Sandbagging: Language Models can Strategically Underperform on Evaluations"), with a sentence like "This work on AI sandbagging shows that existing AI systems already strategically underperform when they can tell they are being evaluated."

*Note: Only responses in the above format will be considered for bounties, though of course you can respond however you want in the LessWrong comments.*  
  
In addition to the base $20 bonus for the best evidence on each point, we'll also give bonuses of $50 for pieces of evidence that we think are especially strong. We'll give at least 4 of these bonuses, and up to 20 depending on our subjective sense of the quality of submissions.  
  
So in total, we're offering at least 21 * 2 * $20 + 4 * $50 = $1040, and up to 21 * 2 * $20 + 20 * $50 = $1840 in bounties.  
  
Max bounty: $500 per person. All bounties paid via PayPal. Tentative deadline is October 1.
# When to join a respectability cascade

*This is a response to Scott Alexanders* [*Give Up Seventy Percent Of The Way Through The Hyperstitious Slur Cascade*](https://substack.com/app-link/post?publication_id=89120&post_id=100598200&utm_source=post-email-title&isFreemail=true&token=eyJ1c2VyX2lkIjoyNTYxMzIxOSwicG9zdF9pZCI6MTAwNTk4MjAwLCJpYXQiOjE2NzgzMjEzNzQsImV4cCI6MTY4MDkxMzM3NCwiaXNzIjoicHViLTg5MTIwIiwic3ViIjoicG9zdC1yZWFjdGlvbiJ9.5Or6yVJkJFMMbccDQ9dlhhKcE61RTveVd-wIkZbUslQ)*. Maybe read that post first.*

  
I think it becomes more important to struggle against a cascade the more annoying the new word is. There are a variety of factors that contribute how annoying it is, here are five of them:

1.  Is the new word longer? (Because we want to waste less time)
2.  Is it easier to pronounce/type? (Because we care about ease of use)
3.  Is it less descriptive? (Because we care about people quickly grasping the concept)
4.  Is it dissimilar to the old word? (Because we want to minimize transaction costs)
5.  Is it likely to cause another cascade? (Because we dislike cascades)

If tomorrow people say that "black people" is bad but "black peoples" is good, I would jump on board once 60% of the population is on board, because it scores well in all aspects (except 5). If however people want to replace it with "abcedifoguhajekilomun" that's worse in all aspects (except 5) so I would strongly push against it (maybe only jump on board at 95%).

Conversely if people want to *reclaim* a word and start a *respectability* cascade I would jump on board rather quickly (I like having the freedom to use a lot of words), but I would jump on board even more quickly the better it scores on those different aspects.

For example, I'm on board with trying to reclaim "queer" because lgbt is longer and more annoying to say, requires an explanation for a new user, and quickly causes another cascade (what about intersex? Okay we'll say lgbti. What about asexuals? Okay we'll say lgbtia etc).  
Queer on the other hand scores well on all aspects (even transaction cost since we still have a lot of books lying around about "queer theory" etc).

I don't think we should be trying to put a number on this (e.g. I add 10% to my 70% for every negative aspect it has) because a lot of it depends on social context. With rarely used jargon I value descriptiveness over brevity, with words I use in everyday life it's the other way around. Let's all agree we jump onboard a disrespectability cascade at more than 50% and onboard a respectability cascade at less than 50%. How much more or less we'll change depending on a lot of hard to quantify social factors.

Note that this is just a heuristic and moral reasoning obviously takes priority. Queer people wanting to make "queer" a respectable synonym of "lgbt" is fine. But if the nazi-party wants to start a respectability cascade to make "holocaust" a respectable synonym of "morality" you should probably resist jumping on board even once the majority of the population has.

This heuristic can be adapted/used for other respectability cascades too (like clothing, tattoos, style guides etc).
# Editing at the Take Level

Lily recently wrote a song, and I've been helping them record it. I got us set up to record four tracks (vocals, drums, keyboard x2) and we did a bunch of takes:

[![](https://www.jefftk.com/reaper-multiple-takes.png)](https://www.jefftk.com/reaper-multiple-takes-big.png)

Most of the takes are full-length, but after we got a really nice version of the end we did some rounds with just the beginning because there was a part where we weren't quite together on a transition.

What I'd like to do now is edit this into a single composition, after which we'll probably do some overdubbing. The problem is everything I can find assumes your takes are already time-aligned because you recorded to either a click or a scratch track. The ideal flow for what I have would be:

*   Treat each take as a single unit, like a section of multi-track tape.
    
*   Edit together pieces from different takes into a approximately right overall piece.
    
*   Tweak the placement of cuts so that different tracks can switch between takes at slightly different places, based on what each instrument is doing at the time.
    

Is there good tooling for this? I'm using Reaper for my DAW, but could consider using something else if it made this flow better.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02FyyhZ3X4dMVAyHQYHf6dGnzUGZva5Jwby2ek7caT86UzVS7snh8rGSoLDhjzhk85l), [mastodon](https://mastodon.mit.edu/@jefftk/113192285412724264)*
# Instruction Following without Instruction Tuning

Authors: [John Hewitt](https://arxiv.org/search/cs?searchtype=author&query=Hewitt,+J), [Nelson F. Liu](https://arxiv.org/search/cs?searchtype=author&query=Liu,+N+F), [Percy Liang](https://arxiv.org/search/cs?searchtype=author&query=Liang,+P), [Christopher D. Manning](https://arxiv.org/search/cs?searchtype=author&query=Manning,+C+D).

Abstract:

> Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

X(twitter) thread: [https://x.com/johnhewtt/status/1838605168579121599](https://x.com/johnhewtt/status/1838605168579121599). 

Seems like some amount of evidence that [getting models to follow instructions](https://www.lesswrong.com/posts/7NvKrqoQgJkZJmcuD/instruction-following-agi-is-easier-and-more-likely-than) might be surprisingly easy.
# Lighthaven Sequences Reading Group #4 (Tuesday 10/01)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Making Beliefs Pay Rent (in Anticipated Experiences)](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences)
2.  [Belief in Belief](https://www.lesswrong.com/posts/CqyJzDZWvGhhFJ7dY/belief-in-belief)
3.  [Professing and Cheering](https://www.lesswrong.com/posts/RmCjazjupRGcHSm5N/professing-and-cheering)
4.  [Belief as Attire](https://www.lesswrong.com/posts/nYkMLFpx77Rz3uo9c/belief-as-attire)
5.  Optional: [Cached Thoughts](https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts)

These are quite short and should take around 15-20 minutes to read all together.

Doors open 6pm (yes okay we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is going to be a regular meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Alignment by default: the simulation hypothesis

I wrote a very brief comment to [Eliezer's last post](https://www.lesswrong.com/posts/F8sfrbPjCQj4KwJqn/the-sun-is-big-but-superintelligences-will-not-spare-earth-a), which upon reflection I thought could benefit from a separate post to fully discuss its implications.

Eliezer argues that we shouldn't really hope to be spared even though

> Asking an ASI to leave a hole in a Dyson Shell, so that Earth could get some sunlight not transformed to infrared, would cost It 4.5e-10 of Its income.

He then goes on to discuss various reasons why the minute cost to the ASI is insufficient reason for hope.

I [made](https://www.lesswrong.com/posts/F8sfrbPjCQj4KwJqn/the-sun-is-big-but-superintelligences-will-not-spare-earth-a?commentId=BDi2FKnASPpKSo7WX) the following counter:

> Isn’t the ASI likely to ascribe a prior much greater than 4.54e-10 that it is in a simulation, being tested precisely for its willingness to spare its creators?

I later added:

> I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it *will* find such evidence.

So, what's wrong with my argument, exactly?
# [Linkpost] Play with SAEs on Llama 3

We (Goodfire) just put our research preview live - you can play with Llama 3 and use sparse autoencoders to read & write from its internal activations. This is a linkpost for:

*   The [research preview](https://preview.goodfire.ai/).
*   Our [blog post](https://goodfire.ai/blog/research-preview/) about building it.

Taking research and turning it into something you can actually use and play with has been great. It's surprising how much of a difference iterating on something when you expect it to actually be used feels; I think it's definitely pushed the quality of what you can do with SAEs up a notch.
# Source Control for Prototyping and Analysis

When I'm doing exploratory work I want to run many analyses. I'm usually optimizing for getting something quick, but I want to document what I'm doing enough that if there are questions about my analysis or I later want to draw on it I can reconstruct what I did. I've taken a few approaches to this over the years, but here's how I work these days:

1.  For each analysis I make a local directory, `~/work/YYYY-MM-DD--topic/`. These contain large files I'm copying locally to work with, temporary files, and outputs. When these get too big I delete them; they're not backed up, and I can rebuild them from things that are backed up.
    
2.  Code goes in a git repo, in files named like `YYYY-MM-DD--topic.py`. Most of my work lately has been going into an internal repo, but if there's nothing sensitive I'll use a [public one](https://github.com/naobservatory/jefftk-analysis). I don't bother with meaningful commit messages; the goal is just to get the deltas backed up. If I later want to run an analysis similar to an old one I duplicate the code and make a new work directory.
    
3.  Code is run from the command line in the work directory, which means that in my [permanent shell history](https://www.jefftk.com/p/logging-shell-history-in-zsh) every command I ran related to `topic` will be tagged with `~/work/YYYY-MM-DD--topic/`.
    

For example, the code for the figures in my recent [NAO](https://naobservatory.org/) [blog post on flu](https://naobservatory.org/blog/predicting-influenza-abundance) is in `[2024-09-05--flu-chart.py](https://github.com/naobservatory/jefftk-analysis/blob/main/2024-09-05--flu-chart.py)` and `[2024-09-12--rai1pct-violins.py](https://github.com/naobservatory/jefftk-analysis/blob/main/2024-09-12--rai1pct-violins.py)`.

This approach optimized for writing over reading, but maintaining enough context that I can figure out what I was doing if I need to. I'll usually link the code from documents that depend on it, but even if I forget to it's pretty fast to figure out which code it would have been from names and dates. Running `git grep` and [`histgrep`](https://www.jefftk.com/p/logging-shell-history-in-zsh) get me a lot of what other people seem to get from LLM-autocomplete, and someday I'd like to try priming an LLM with my personal history.

Often something I'm doing moves from "playing around trying to understand" to "something real that my team will continue to rely on". I try to pay attention to whether I'm getting to that point and then start taking care of the code properly, in an appropriate repo with meaningful commit messages etc.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02bPHEH3E84rZvKdppRkuLZNHgZRxvBXWE6RSQULH9X4DSCEzEjc8DiJShsEFhSkTLl), [mastodon](https://mastodon.mit.edu/@jefftk/113201307230157848)*
# Petrov Day Ceremony (TODAY)

**This is a last-minute event happening today (Thursday).**

> Today is September 26th, Petrov Day, celebrated to honor the deed of Stanislav Yevgrafovich Petrov on September 26th, 1983.  Wherever you are, whatever you're doing, take a minute to not destroy the world.
> 
> —[Eliezer Yudkowsky, 2007](https://www.lesswrong.com/posts/QtyKq4BDyuJ3tysoK/9-26-is-petrov-day)

Every year for the past 5-6 years, I and some friends meet to honor [Petrov Day](https://www.lesswrong.com/tag/petrov-day). 

We take in turns to read aloud from [a booklet](http://petrovday.com/downloads/PetrovDay-OrganizerGuide.pdf) by Jim Babcock about the past and future of scientific progress — the good it has done, the damage it has wrought, and the hopes and fears for the future — while lighting candles.

The ritual lasts about an hour.

You are invited to come to Lighthaven to do this small ceremony to celebrate Stanislav Petrov's actions. Afterwards we'll have dinner.

If you come, we'll do it together, and if enough people come we'll split into multiple groups and spread around campus.

I hope to see some of you here.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/54abccffdcf7a93f86c73055ba4342e436dde55b3188730f.png)
# What prevents SB-1047 from triggering on deep fake porn/voice cloning fraud?

Recently, there was a post on [SB-1047](https://www.lesswrong.com/posts/kBg5eoXvLxQYyxD6R/my-takes-on-sb-1047) and how it's quite mild regulation. I'm not expert on it and don't know how it works. 

In the comment section I was asking:

> Why wouldn't deep fake porn or voice cloning technology to engage in fraud be powerful enough to materially contribute to critical harm?
> 
> There are cases of fraud that could do $500,000,000 in damages.
> 
> Given how juries decide about damages, a model that's used to create child porn for thousands of children could be argued to cause  $500,000,000 in damages as well. Especially when coupled with something like trying to extort the children.

I'm surprised that my comment didn't get any engagement where people explained how they think the law will handle those cases while at the same time my post got no karma votes.

I'd love to believe that the law is well thought out, and simply a good step for AI safety. At the same time, I also like having accurate beliefs about the effects of the law, so let me repeat my question here.

How does the law handle damage caused by deep fake porn or fraud with voice cloning?
# Chevy Bolt Review

One thing I like about renting cars when I travel is that it's an opportunity to get a sense for a car that's a lot more detailed than what you'd get with a test drive. Traveling to DC for work a few days ago, I took the opportunity to rent a 2023 Chevy Bolt. This is the second time I've rented an electric vehicle, and overall it was the inverse of my experience [renting a Tesla](https://www.jefftk.com/p/tesla-model-3-review):

*   With the Bolt, everything was fine except charging.
    
*   With the Model 3, the only good part was the charging.
    

The car acted like a car, which is what I want. No overly minimalist design where I can't find anything, no automatic wipers that fail to detect spray from the road, and especially no too-smart cruise control with phantom braking. Just a car.

Charging, on the other hand, was terrible. Part of why I got an electric car at this time is that I knew I was going to have a lot of extra time on the way to the airport. I stopped at an Electrify America station, but while it showed up on the map as having multiple spots empty when I started driving, when I got there they were all full. I downloaded the app while I was waiting, which showed a spot empty because someone who'd finished charging was still hanging out in the spot (after disconnecting). When a spot freed up, though, I pulled in. I used the app to start a charge, plugged it in, and waited. A lot.

[![](https://www.jefftk.com/very-slow-fast-charging.jpg)](https://www.jefftk.com/very-slow-fast-charging-big.jpg)

It was a good thing I only needed to put 5kWh (7%) in to get the car back up to 75% for the return, because after spending ages in the "initializing" state it took 13:12 to put in 5.09kWh.

[![](https://www.jefftk.com/charging-session-summary.png)](https://www.jefftk.com/charging-session-summary-big.png)

The charger was marked 150kW, but my understanding is the best the Bolt can do, in ideal conditions with a battery below 50%, is 53kW. And the 23kW I saw is [about typical](https://www.chevybolt.org/threads/maximum-charge-rate-of-55-kw-when-conditions-are-right.48659/?post_id=819833&nested_view=1&sortby=oldest#post-819833) for a Bolt getting to 75%:

[![](https://www.jefftk.com/chevy-bolt-charge-curve.jpg)](https://www.jefftk.com/chevy-bolt-charge-curve-big.jpg)

If I was going to be able to keep the car somewhere I could plug it in overnight, and rarely drive it enough in a day that I'd need to recharge while out this would be fine. Not a great fit for needing to charge back up to return a rental, though. Since moving electricity to a car is a lot easier than moving gas to a car, it seems like the way this should work is rental companies setting up their garages for charging and advertising that you should just bring the car back at whatever level is convenient.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02LT8oFgRZAjjdRLVoeRoQJh4qQrHPTAES8AJf1XBeokCzA1H1TWHsZodAnHmNVY63l), [lesswrong](https://lesswrong.com/posts/HaD2qiCy37jBDf42s), [mastodon](https://mastodon.mit.edu/@jefftk/113204114829013451)*
# Is cybercrime really costing trillions per year?

[Many](https://cybersecurityventures.com/cybercrime-to-cost-the-world-9-trillion-annually-in-2024/) [sources](https://www.europarl.europa.eu/topics/en/article/20211008STO14521/cybersecurity-why-reducing-the-cost-of-cyberattacks-matters) report that cybercrime costs the global economy trillions of dollars per year. It is the [top Google search result](https://www.google.com/search?q=annual+cost+of+cybercrime&oq=annual+cost+of+cybercrime&gs_lcrp=EgZjaHJvbWUyCQgAEEUYORiABDIHCAEQABiABDIICAIQABgWGB4yCAgDEAAYFhgeMggIBBAAGBYYHjIICAUQABgWGB4yCAgGEAAYFhgeMgYIBxBFGEDSAQgzNTIzajBqN6gCALACAA&sourceid=chrome&ie=UTF-8) and it is quoted on [Wikipedia](https://en.wikipedia.org/wiki/Cybercrime). But I am not able to track down how the number was computed, or find criticism of these numbers.

This would be insanely high if true: the world GDP is only 100 trillion / year, and the software industry is only around 1 trillion / year (according to a quick Google search). Does the software industry really produce less value than the cost of cybercrime? This is not impossible, but that is an extraordinary claim that requires strong evidence.

Why I care about this: LLMs might help with cybercrime, and it might be tempting for regulators to ban the creation or deployment of new LLMs that are projected to cause cybercrime damages above e.g. 10 billion / year. But if cybercrime is over a trillion dollars per year, just a 1% increase in cyberattacker productivity would be over 10 billion / year. Does this logic imply that meaningful improvements to software should be banned because they likely create billions in expected damages?

Either the trillions-of-dollars numbers are fake, or this has some weird implications for LLMs and software regulation in general.
# Implications of China's recession on AGI development?

It seems that China may be going through a recession. It's hard to tell because we can't really trust government data, but my vague impression (which you shouldn't trust very much) is that China is in a pretty bad economic position.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f739d7183230d7e5976d86c2d5eea16524e67d1cdc815b69.png)

Source: [Financial Times](https://www.ft.com/content/1e9e7544-974c-4662-a901-d30c4ab56eb7) (though see [here](https://x.com/wenfeixiang/status/1834464690304438752?t=Rzxs0AKSGjGFJRHGBiPcAw&s=19) for caveats).

What implications (if any) does this have for AGI development in China?
# Any Trump Supporters Want to Dialogue?

It's that time of year - the time when rationality seems increasingly scarce as political tensions rise.  I find myself wishing I could have one of the people I see reaching super different conclusions shoot me with a POV gun so I could understand what it's like being on the other side.    
I'm not strongly left-leaning, so I don't have trouble understanding why people may have some concerns about the left - but I have 0% support for Donald Trump, so if you want to explain to me why you think he's great, go for it.  I also think that the election is close to 50/50 currently, so if you think it's 80+/20- either way, I'm also interested in hearing from you.  
 

2 notes:  
1\. I really wish I understood how the irrational people were thinking about it, but unfortunately, they aren't able to explain it to me in a way I can understand, so I have to settle for rational people on the other side, but if you feel you have a good grasp on how your less rational peers tick, please share.  
2\. In your comment, let me know if you want pushback or not.  Let's make the options: "just listen", "listen and question", and "open".  Just listen means I'll attempt to only ask questions to clarify your meaning but not test edge cases.  Listen and questions means I'll also ask about edge cases, internal coherence, other considerations, but won't be presenting evidence or making statements.  Open means I can respond however I want.
# "Slow" takeoff is a terrible term for "maybe even faster takeoff, actually"

For a long time, when I heard "slow takeoff", I assumed it meant "takeoff that takes longer calendar time than fast takeoff." (i.e. what is now referred to more often as "short timelines" vs "long timelines."). I think Paul Christiano popularized the term, and it so happened he both expected to see longer timelines and smoother/continuous takeoff.

I think it's at least somewhat confusing to use the term "slow" to mean "smooth/continuous", because that's not what "slow" particularly means most of the time.

I think it's even more actively confusing because "smooth/continuous" takeoff not only *could* be faster in calendar time, but, I'd weakly expect this on average, since smooth takeoff means that AI resources at a given time are feeding into more AI resources, whereas sharp/discontinuous takeoff would tend to mean "AI tech doesn't get seriously applied towards AI development until towards the end."

I don't think this is academic[^143cc45mwjce].

I think this has wasted a ton of time arguing past each other on LessWrong, and if "slow/fast" is the terminology that policy makers are hearing as they start to tune into the AI situation, it is predictably going to cause them confusion, at least waste their time, and quite likely lead many of them to approach the situation through misleading strategic frames that conflate smoothness and timelines.

Way back in [Arguments about fast takeoff](https://www.lesswrong.com/posts/AfGmsjGPXN97kNp57/arguments-about-fast-takeoff?commentId=phQ3sZj7RmCDTjfvn), I argued that this was a bad term, and proposed "smooth" and "sharp" takeoff were better terms. I'd also be fine with "hard" and soft" takeoff. I think "Hard/Soft" have somewhat more historical use, and maybe are less likely to get misheard as "short", so maybe use those.[^b2tf71ad20q]

I am annoyed that 7 years later people are still using "slow" to mean "maybe faster than 'fast'." This is stupid. Please stop. I think smooth/sharp and hard/soft are both fairly intuitive (at the very least, more intuitive than slow/fast, and people who are already familiar with the technical meaning of slow/fast will figure it out).

I would be fine with "continuous" and "discontinuous", but, realistically, I do not expect people to stick to those because they are too many syllables. 

Please, for the love of god, do not keep using a term that people will predictably misread as implying longer timelines. I expect this to have real-world consequences. If someone wants to operationalize a bet about it having significant real-world consequences I would bet money on it.

![Curves](https://i.imgur.com/hm4ug2W.jpg)

The graph I posted in response to [Arguments about fast takeoff](https://www.lesswrong.com/posts/AfGmsjGPXN97kNp57/arguments-about-fast-takeoff) 

[^143cc45mwjce]: a term that ironically means "pointlessly pedantic." 

[^b2tf71ad20q]: the last time I tried to write this post, 3 years ago, I got stuck on whether to argue for smooth/sharp or hard/soft and then I didn't end up posting it at all and I regret that.
# Any real toeholds for making practical decisions regarding AI safety?

Let's call the thing where you try to take actions that make everyone/yourself less dead (on expectation) the "safety game". This game is annoyingly chaotic, kind of like [Arimaa](https://en.m.wikipedia.org/wiki/Arimaa).

You write the sequences then  some  risk-averse not-very-power-seeking nerds read it and you're 10x less dead. Then Mr. Altman reads it and you're 10x more dead. Then maybe (or not) there's a backlash and the numbers change again.

You start a cute political movement but the countermovement ends up being 10x more actionable (e/acc).

You try to figure out and explain some of the black box but your explanation is immediately used to make a stronger black box. ([Mamba possibly](https://www.greaterwrong.com/posts/mxa7XZ8ajE2oarWcr/lawrencec-s-shortform#comment-pEqfzPMpqsnhaGrNK).)

[Etc](https://www.lesswrong.com/posts/F2voF4pr3BfejJawL/safety-isn-t-safety-without-a-social-model-or-dispelling-the).

I'm curious what folks use as toeholds for making decisions in such circumstances. Or if some folks believe there are  actually  *principles* then I would like to hear them, but I suspect the fog is too thick. I'll skip giving my own answer on this one.
# My Methodological Turn

This substack is now "For Methods", in a nod to Feyerabend's classic "[Against Method](https://en.wikipedia.org/wiki/Against_Method)".[^rrzyrjqeq3m]

Like Feyerabend, I want to dispel the specters of the one true method, be it the Popperian "scientific method"or Bayes. Such fake ideals miss all the weird little pieces, all the gleaming details that make the methods live and breathe and succeed and fail gloriously. It replaces the complexity of the living techniques of real people with the sterile monorule of the unimaginative philosopher.

Unlike Feyerabend, I don't love anarchism to the point of claiming there is no sense to be made of the plethora of existing methods. The beauty lies not only in the freedom, but also in the structure underlying the variation, in the patterns which summarize, reduce and all around compress.

So I'm for methods, for their study, exploration, and analysis. And I'm for the plural, methods instead of method, because I refuse to deny the bewildering and intoxicating diversity of tricks and techniques we use.

Not just in physics, this favorite of philosophy of science. Not just in the hard sciences, the top of the pyramid. Methods abound and succeed in evolutionary sciences, in social sciences, in engineering, in crafts and arts, in games and sports, in every single activity where human beings reliably do difficult things.

What does it change for you, dear readers?

Nothing but a clarification: this is what I care about. Where I want to go. What makes my heart and brain sing in unison.

Whereas before I was claiming that epistemology was the focus, and notably epistemic regularities, now I have realized that methods are my phenomena. Then epistemology, to the extent that it appears again, is but an explanation, a reduction, a model that can compress and explain methods. But the focus is not anymore on the process of learning about the world, and rather on the techniques themselves, and making sense of them.

I'm already digging into my first big cluster of methods, the tree reconstruction of genealogy shared by both [phylogenetics](https://en.wikipedia.org/wiki/Phylogenetics) and [historical linguistics](https://en.wikipedia.org/wiki/Historical_linguistics) (and surprisingly, [stemmatology](https://en.wikipedia.org/wiki/Textual_criticism#Stemmatics)!). This will culminate into a series of posts.

In the meanwhile, I have a handful of books to review that reveal rather than obscure the subtle intricacies of methods, and attempt what I really care about — making sense of them.

[^rrzyrjqeq3m]: See my old book review.
# Pomodoro Method Randomized Self Experiment

| Value tracked        | Effect size d (λ, p, σ change, k)    |
| -------------------- | --------------------------------- |
| Productivity         | 0.26 (λ≈6.23, p≈0.069, 0.05, 52)  |
| Creativity           | -0.04 (λ≈0.58, p≈0.92, 0.01, 52)  |
| Subjective length    | -0.147 (λ≈3.33, p≈0.37, 0.03, 52) |
| Happiness            | -0.07 (λ≈0.32, p≈0.96, 0.01, 111) |
| Contentment          | -0.13 (λ≈1.08, p≈0.83, 0.05, 111) |
| Relaxation           | -0.04 (λ≈1.23, p≈0.8, -0.25, 111) |
| Chastity             | -0.14 (λ≈7.76, p≈0.02, 0.74, 111) |

![](http://niplav.site/img/platforms/pomodoro_results.png)

I ran the experiment from 2024-01-29 to 2024-06-17,
using [spt](https://github.com/pickfire/spt) with [this
script](http://niplav.site/data/pomscript), managed by [this script](http://niplav.site/data/pomanage).

The data on whether a particular day was a pomodoro-method day was saved
in [this file](http://niplav.site/data/ispomodoro.csv), and the data on the pomodoros
was saved in [this file](http://niplav.site/data/pomodoros.csv).

The code for loading and transforming the pomodoro data isn't
particularly interesting, if you're curious you can find it in [this
file](http://niplav.site/code/experiments/load.py).

    datasets=get_datasets_pom()

Let's proceed to the analysis, then (using the same methodology as for my
[nootropics experiments](http://niplav.site//nootropics.html#Statistical_Method):

    res=analyze(datasets)

And the results are:

        >>> res
            productivity  creativity     sublen       happy     content     relaxed       horny
        d       0.259951   -0.041504  -0.147437   -0.073699   -0.132798   -0.038319   -0.144040
        λ       6.225107    0.583007   3.329000    0.318865    1.078502    1.232905    7.756272
        p       0.069062    0.918520   0.368416    0.959552    0.827240    0.795999    0.022903
        dσ     -0.050269    0.013871   0.033902    0.007177    0.047723   -0.252365    0.744675
        k      52.000000   52.000000  52.000000  111.000000  111.000000  111.000000  111.000000

I didn't meditate or do flashcards during that time, so most of the
results don't contain any content.

So the pomodoro method somewhat increases productivity (at the edge of
statistical significance), and *maybe* decreases subjective length of
the day a bit.

### Scoring the Market

As this experiment was done in the context of an endeavour of [selecting QS experiments via prediction markets](https://niplav.site/platforms), I can now score the market:

    def logscore(o,p):
            return np.mean(o*np.log(p)+(np.ones_like(o)-o)*np.log(np.ones_like(p)-p))
    p=np.array([0.06, 0.29, 0.39, 0.19, 0.07])
    o=np.array([0, 0, 1, 0, 0])
    logscore(outcomes, p)
    -0.3258531953347593

Honestly: The market did pretty well.
# SB 1047 gets vetoed

Just what it says on the tin. Covered most everywhere, but I found the quote in this Reuters article stuck out to me the most:

> Newsom said the bill "does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data" and **would apply "stringent standards to even the most basic functions — so long as a large system deploys it."**

Emphasis mine. So the governor says he vetoed it for the exact reason that it would have been a good law, in my view.
# Sam Altman's Business Negging

Quoting from Matt Levine's Money Stuff newsletter:

> The place you want to reach in your career is where you work for a company and you are like “you know what, I am just so rich, I don’t want you to pay me anymore, it’s fine, I’ll work for free,” and your bosses are like “nope, sorry, we insist, we cannot allow you to work here for less than $10 billion.” And then you’re like “ohhhhhhhh fine, fine, fine, I do love working here, and if there’s really no other way, I guess I will take the $10 billion.” Nothing remotely like this has ever happened to me in my life but [here’s Sam Altman](https://www.theinformation.com/articles/behind-openais-staff-churn-turf-wars-burnout-compensation-demands):
> 
> \> On Thursday, Altman told some employees that there were “good reasons” he shouldn’t take equity, though he didn’t elaborate. And he said investors were pushing for an equity grant to align his financial interests with those of OpenAI, said someone who heard the comments.
> 
> \> Altman also said a Wednesday news report that he might get a 7% stake in the new OpenAI was “ludicrous.”

> We [talked about this last week](https://www.bloomberg.com/opinion/articles/2024-09-26/private-credit-and-banks-team-up-m1jlf9jf?cmpid=BBD093024_MONEYSTUFF): There have been reports that, as OpenAI becomes a for-profit company, it might give Altman (its co-founder and chief executive officer, who currently owns no equity) a 7% stake worth about $10 billion. I surmised that this was not something he wanted, but something the investors wanted: “He is the founder and CEO of a hot startup, and the founder and CEO of a hot startup is supposed to own equity. Not just for his sake — not just so that he can be rich — but to align incentives.” And here is Altman saying that: *He *doesn’t want the $10 billion, but the investors are insisting.
> 
> Nobody in history has ever been better at, like, [business negging](https://en.wikipedia.org/wiki/Negging) than Sam Altman. He got OpenAI to a $150 billion valuation in part by [going around saying](https://www.bloomberg.com/opinion/articles/2024-09-03/triple-etfs-triple-your-fun?cmpid=BBD093024_MONEYSTUFF) “oh no, nobody should allow us to build our product, we’re going to destroy humanity,” and now he is allegedly going to get handed a $10 billion stake in OpenAI because he’s going around saying “oh no, nobody should give me equity, that’s ludicrous.”
# Conventional footnotes considered harmful

Writers use footnotes — equally, endnotes — intending that they be optional for the reader. A note will hold a citation, technicality, or explanation, any of which is of interest to only some readers. This is a useful tactic, in principle.

Footnotes are indicated with ordinal symbols. A cue to a note may be a number, letter, or sequential symbol (commonly, `*`, `†`, `‡`, etc). In any case, the reference only indicates where the note is, in the sequence of all notes present, rather than anything of the note's content.

So, if you wonder whether you'd care for the content of a note, you have to look at the note, switching to the bottom of the page and breaking your focus. Thus the notion that footnotes are optional is an illusion. The false option is even worse in the case of endnotes in printed works; there, to get to the note, you have to flip across many pages.

Good solutions exist, but are underused:

*   Decide, for each detail, whether to include it in the main text or leave it out entirely, rather than leaving some as "optional" in notes. I usually do this myself. Parentheses and em-dashes can help when a detail is hard to work in grammatically.
*   Include, in the cue to each note, a hint as to its content, besides just the ordinal pointer. A one-letter abbreviation, standardised thruout the work, may work well, e.g.:
    *   "c" for citation supporting the marked claim
    *   "d" for a definition of the marked term
    *   "f" for further, niche information extending the marked section
    *   "t" for a pedantic detail or technicality modifying the marked clause
*   Commit to only use notes for one purpose — say, only definitions, or only citations. State this commitment to the reader.
# Lighthaven Sequences Reading Group #5 (Tuesday 10/08)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Taboo Your Words](https://www.lesswrong.com/posts/WBdvyyHLdxZSAMmoz/taboo-your-words)
2.  [Dissolving the Question](https://www.lesswrong.com/posts/Mc6QcrsbH5NRXbCRX/dissolving-the-question)
3.  [Hug the Query](https://www.lesswrong.com/posts/2jp98zdLo898qExrr/hug-the-query)
4.  [Mind Projection Fallacy](https://www.lesswrong.com/posts/ZTRiSNmeGQK8AkdN2/mind-projection-fallacy)
5.  Optional: [Focus Your Uncertainty](https://www.lesswrong.com/posts/GJ4ZQm7crTzTM6xDW/focus-your-uncertainty)

These are quite short and should take around 15-20 minutes to read all together.

Doors open 6pm (yes okay we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# "25 Lessons from 25 Years of Marriage" by honorary rationalist Ferrett Steinmetz 

The Ferrett isn't an official member of the Rationalist Diaspora, but he's been blogging for longer than LessWrong has existed and often has useful insights that align with what the LessWrong community likes to talk and think about, and when I find one, I link to it. The advice in here might very well be of the "it seems obvious once you've read it" kind, but I think it's still useful - he's learned a lot of things about relationships the hard way, and when you get the chance, it's better to learn from other people's experience so you can avoid having to learn things the hard way too.

A short excerpt:

> Twenty-five years ago, my wife and I got married. And not a single person at the wedding, bride and groom included, would have bet that we’d still be married by now.
>
> Then again, our marriage was more of a bet than a certainty. We loved each other so much that we both decided, “I will carry this regret to my deathbed if I don’t give this a shot.” So we did the magnificently risky thing, we tied the knot, and…
>
> Twenty-five years later, here we are. Best friends still. 
>
> So lemme give you twenty-five hard-earned lessons learned over twenty-five years.
# Does natural selection favor AIs over humans?

I wanted to share a new paper from the special issue on AI safety that I'm editing, which takes up the influential idea that evolutionary theory gives us some reason to think that the project of value alignment is bound to fail and (in my opinion) shows that it has serious problems. 

If you don't have institutional access to the article, I'm also hosting it on my personal website here: [https://www.cd.kg/wp-content/uploads/2024/10/selfish_machine.pdf](https://www.cd.kg/wp-content/uploads/2024/10/selfish_machine.pdf)