Harley-ml/lesswrong huggingface dataset:
# Apply to a small iteration of MLAB to be run in Oxford

TLDR: We’re running a small iteration of MLAB (~10 participants) in Oxford towards the end of September. If you’re interested in participating, apply [here](https://forms.gle/QV9z6cxNGnS1UJzq6) by 7 September. If you’re interested in being a TA, please email us directly at [oxfordmlab@gmail.com](mailto:oxfordmlab@gmail.com)

*Edit: The dates are now confirmed for the 23 September- 7 October. *

**Background**

MLAB is a [program](https://www.redwoodresearch.org/mlab), originally designed by Redwood Research, to help people upskill for alignment work. We think it’s a good use of time if you want to eventually get into technical alignment work, or if you want to work on theoretical alignment or related fields and think understanding ML would be useful. The program we’re running is slightly shorter than the full MLAB—two weeks instead of three. We’ve condensed the curriculum similarly to how WMLB was condensed last year. 

We plan to have just under 10 participants, and 2-3 TAs.

**Curriculum**

This curriculum might change slightly. Depending on participant interest, we might also have two optional days before the course to work through prereqs (the W0 materials) together.

W0D1 - pre-course exercises on PyTorch and einops (CPU)  
W1D1 - practice PyTorch by building a simple raytracer (CPU)  
W1D2 - build your own ResNet (GPU preferred)  
W1D3 - build your own backpropagation framework (CPU)  
W1D4 - model training Part 1: model training and optimizers (CPU) Part 2: hyperparameter search (GPU preferred)  
W1D5 - GPT Part 1: build your own GPT (CPU) Part 2: sampling text from GPT (GPU preferred)  
W2D1&2 - transformer interpretability (CPU)  
W2D3 - transformer interpretability on algorithmic tasks (CPU)  
W2D4 - intro to RL Part 1: multi-armed bandit (CPU) Part 2: DQN (CPU)  
W2D5 - policy gradients and PPO (CPU) 

Other activities will include guest speakers and reading groups.  

**Logistics**

Dates: 23 September - 7 October

Location: Oxford

Housing will be covered for participants not already living in Oxford.

Travel from within the UK is covered. Travel from outside the UK is not covered.

**Questions**

Feel free to comment questions below or DM any of us.
# Trying a Wet Suit

I get cold very quickly in the water, enough that unless it's close to body temperature I get chilled through within ~15min. This mostly wasn't a problem, because I'd take a quick dip to cool off and then hang out on the beach, but now that I have kids they (and I) want lots of swimming together time. When I touched on this [a few weeks ago](https://www.jefftk.com/p/weekday-evening-beach-picnics) [people recommended](https://www.lesswrong.com/posts/4CmAtozcAHXbakWAz#cBe5BHcz2bauvYaPr) [trying a wetsuit](https://www.lesswrong.com/posts/4CmAtozcAHXbakWAz/weekday-evening-beach-picnics?commentId=r4ENMYrvNECNKWs7A), and yesterday evening I did for the first time!

It was different in a bunch of ways, but on balance I like it a lot. Some things I noticed:

*   Initially there was some air in my suit, which felt funny bubbling out.
    
*   The suit is slightly buoyant, taking some getting used to.
    
*   It still felt cold getting in, until my body had time to heat up the trapped layer of water.
    
*   I didn't get cold in the water! This was with maybe ~78F water and ~82F air. I could play with my kids until they wanted to get out of the water.
    
*   I bought separate pants and a vest, which I wore under my swimsuit. The pants worked very well, and the vest worked ok: I got a bit of water rotating through where they met, and I occasionally needed to pull the vest down. Possibly a full body suit would have been better? But those seem more annoying to get in and out of, and the amount of water moving through was pretty low with the kind of playing we were doing.
    
*   Once I got out of the water, I stayed wet a lot longer. This was warmer than usual for the first part, since it was a warm sort of wet, though after we got to the stage where I normally would have been fully dry (~45m?) I was mildly colder.
    

Overall I'm happy, and am looking forward to swimming with it again in the future!

(My kids are now asking if they can get ones too, which is fine with me!)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0cQgqdWb9jReg3iVuivb93Xik1f91fJyYaAad2YPo21TEwJAJg2TK4haqY1xajWRal), [mastodon](https://mastodon.mit.edu/@jefftk/110962147402649565)*
# No More Freezer Pucks

A while ago I wrote about how I'd make [freezer pucks](https://www.jefftk.com/p/frozen-lunch-pucks):

[![](https://www.jefftk.com/sauce-pucks-bagged.jpg)](https://www.jefftk.com/p/frozen-lunch-pucks)

These work ok, but there are a couple issues:

*   A large batch requires a lot of muffin tins
*   Cleaning muffin tins is annoying
*   It can be hard to get them out of the tins

I've switched to a different system where I freeze a flat tray in a thin enough layer that I can break it easily:

[![](https://www.jefftk.com/frozen-flat-tray.jpg)](https://www.jefftk.com/frozen-flat-tray-big.jpg)

It takes a bit of practice getting the layer thickness right: generally the higher the water content the thinner it needs to be if you don't want an awkward block.

I put a layer in a freezer bag:

[![](https://www.jefftk.com/one-layer-in-bag.jpg)](https://www.jefftk.com/one-layer-in-bag-big.jpg)

Before putting in additional layers I put down a scrap of parchment paper or wax paper:

[![](https://www.jefftk.com/multiple-layers-in-bag.jpg)](https://www.jefftk.com/multiple-layers-in-bag-big.jpg)

It's easier and faster than the puck method, though with higher risk of accidentally ending up with a block and needing to defrost some.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02FskXYfat79kqGYxNzrUWiQr1EkJngSpYpPfmKZcKTPpS2iqxBd8hDmm4NmLyeWXKl), [mastodon](https://mastodon.mit.edu/@jefftk/110967908701141794)*
# Incentives affecting alignment-researcher encouragement

My hypothesis: I think the incentives for "cultivating more/better researchers in a preparadigmatic field" lean towards "don't discourage even less-promising researchers, because they could luck out and suddenly be good/useful to alignment in an unexpected way".

Analogy: This is like how investors encourage startup founders because they bet on a flock of them, not necessarily because any particular founder's best bet is to found a startup.

If timelines are short enough that [our survival depends on [unexpectedly-good paradigms]], and [unexpectedly-good paradigms] come from [black-swan researchers], then the AI alignment field is probably (on some level, assuming some coordination/game theory) incentivized to [black-swan farm](http://www.paulgraham.com/swan.html) researchers.

Note: **This isn't necessarily bad** (and in fact it's probably good overall), it just puts the incentives into perspective. So individual researchers don't feel so bad about "not making it" (where "making it" could be "getting a grant" or "getting into a program" or...)

The questions: Is this real or not? What, **if anything**, should anyone do, with this knowledge in hand?
# Newcomb Variant

There are two boxes. If you open a box, you get to keep the contents.

Omega is an omniscient oracle. Omega predicts what will happen if you open the first box and observe that it contains $100.
- If Omega predicts that you will open the second box, then Omega puts $0 in both boxes.
- If Omega predicts that you will not open the second box, then Omega puts $100 in both boxes.

You open the first box. It contains $100. Do you open the second box?

# Answer

:::spoiler
No. If you open the second box, that means you're being simulated by Omega. If you're being simulated by Omega, then opening the second box ends the simulation and kills you.
:::

# Extra Credit [Orthogonal Thinking]

You have not yet opened the first box. How do you generate (at least) $250 in profit? [Please use spoiler tags or equivalent obfuscation when discussing the extra credit.]
# Barriers to Mechanistic Interpretability for AGI Safety

I gave a talk at MIT in March earlier this year on barriers to mechanistic interpretability being helpful to AGI/ASI safety, and why by default it will likely be net dangerous. Several people seem to be coming to similar conclusions recently (e.g., [this recent post](https://www.lesswrong.com/posts/LNA8mubrByG7SFacm/against-almost-every-theory-of-impact-of-interpretability-1)). 

I discuss two major points (by no means exhaustive), one technical and one political, that present barriers to MI addressing AGI risk:

1.  **AGI cognition is interactive.** AGI systems interact with their environment, learn online and will externalize massive parts of their cognition into the environment. If you want to reason about such a system, you also need a *model of the environment*. Worse still, AGI cognition is *reflective*, and you will also need *a model of cognition/learning*.
2.  **(Most) MI will lead to capabilities, not oversight.** Institutions are not set up and do not have the incentives to resist using capabilities gains and submit to monitoring and control.

This being said, there are more nuances to this opinion, and a lot of it is downstream of lack of coordination and the downsides of publishing in an adversarial environment like we are in right now.  I still endorse the work done by e.g. Chris Olah's team as brilliant, but extremely early, scientific work that has a lot of steep epistemological hurdles to overcome, but I unfortunately also believe that on net work such as Olah's is at the moment more useful as a safety-washing tool for AGI labs like Anthropic than actually making a dent on existential risk concerns.

[Here](https://docs.google.com/presentation/d/12i93HZ91DfN-gfeP1RUtUPhjzhakOyyfpeQ5RjcUEuY/edit?usp=sharing) are the slides from my talk, and you can find the video [here](https://www.youtube.com/watch?v=wKI9hmaIbpg).
# Loft Bed Fan Guard

I really like ceiling fans (big fan!) and we have them in most of the rooms of our house. About a year ago we also [built loft beds](https://www.jefftk.com/p/building-the-loft-beds) for the older two kids bedrooms:

[![](https://www.jefftk.com/lily-loft-bed-finished.jpg)](https://www.jefftk.com/p/building-the-loft-beds)

Which offers a potential conflict: you don't want people and spinning blades to occupy the same place. While in that picture they look close together, they're not actually that close: when climbing up and down normally there's a good bit of room between you and the fan. But if you turn around while on the ladder and start playing, you can get into the path of the fan.

While our kids are used to this, they also have friends over. We tried having a rule that the fan needs to stay off when friends are over, but it didn't work very well: they'd forget or a friend would flip the switch. After one of Lily's friends got bonked (they're fine) we decided to make a guard for it:

[![](https://www.jefftk.com/fan-guard-on-loft-bed.jpg)](https://www.jefftk.com/fan-guard-on-loft-bed-big.jpg)

The idea is, it's easy to see and avoid a bright colored piece of wood, much more so than an off-white spinning fan blade. And if you're on the right side of the wood then you're away from the fan.

Lily and I made it together: we measured the angle, cut one end on the miter saw, measured again, cut the other, sanded, painted, and put it up. When she got bored I did some of the steps while she played elsewhere. We did ~30% of the work together, which was ~60% of my time on it.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0mk3T4c6BLdSVZWNnmP8ptrxqZH1Du3fkRLuUgQR5K9fFvtMai2fEsyWxu6gJMuXNl), [mastodon](https://mastodon.mit.edu/@jefftk/110973138639657001)*
# Broken Benchmark: MMLU

Phillip over at the AI Explained channel has been running some experiments on his SmartGPT framework against the MMLU benchmark and discovered a not-insignificant amount of issues with the problem set.

Among them:

*   Crucial context missing from questions (apparently copy-paste errors?)
*   Ambiguous sets of answers
*   Wrong sets of answers

He highlights a growing need for a proper benchmarking organization that can research and create accurate, robust, sensible benchmarking suites for evaluating SOTA models.

I found this video to be super interesting and the findings to be very important, so I wanted to spread this here.
# Paper Walkthrough: Automated Circuit Discovery with Arthur Conmy

Arthur Conmy's [Automated Circuit Discovery](https://arxiv.org/pdf/2304.14997.pdf) is a great paper that makes initial forays into automating parts of mechanistic interpretability (specifically, automatically finding a sparse subgraph for a circuit). In this three part series of Youtube videos, I interview him about the paper, and we walk through it and discuss the key results and takeaways. We discuss the high-level point of the paper and what researchers should takeaway from it, the ACDC algorithm and its key nuances, existing baselines and how they adapted them to be relevant to circuit discovery, how well the algorithm works, and how you can even evaluate how well an interpretability method works.
# Long-Term Future Fund Ask Us Anything (September 2023)

The Long-Term Future Fund has an AMA up on the Effective Altruism Forum.

There’s no real deadline for questions, but let’s say we have a soft commitment to focus on questions asked on or before September 8th.

I'd prefer centralizing the questions to one place. If you don't want to post of the forum, for whatever reason (ideological opposition to EA, got banned before, lost your password and don't want to make a new comment), I'm happy to for you to comment here and for me to repost them on the Forum. 

Our donation link is [here](https://www.givingwhatwecan.org/funds/effective-altruism-funds?utm_source=eafunds).

**Related posts**
-----------------

*   [LTFF and EAIF are unusually funding-constrained right now](https://forum.effectivealtruism.org/posts/PAco5oG579k2qzrh9/ltff-and-eaif-are-unusually-funding-constrained-right-now)
*   [EA Funds organizational update: Open Philanthropy matching and distancing](https://forum.effectivealtruism.org/posts/zt6MsCCDStm74HFwo/ea-funds-organisational-update-open-philanthropy-matching)
*   [Long-Term Future Fund: April 2023 grant recommendations](https://forum.effectivealtruism.org/posts/zZ2vq7YEckpunrQS4/long-term-future-fund-april-2023-grant-recommendations)
*   [What Does a Marginal Grant at LTFF Look Like?](https://forum.effectivealtruism.org/posts/7RrjXQhGgAJiDLWYR/what-does-a-marginal-grant-at-ltff-look-like-funding)
*   Asya Bergal’s [Reflections on my time on the Long-Term Future Fund](https://forum.effectivealtruism.org/posts/9vazTE4nTCEivYSC6/reflections-on-my-time-on-the-long-term-future-fund)
*   Linch Zhang’s [Select examples of adverse selection in longtermist grantmaking](https://forum.effectivealtruism.org/posts/sWMwGNgpzPn7X9oSk/select-examples-of-adverse-selection-in-longtermist)
# Cleaning a SoundCraft Mixer

In [Kingfisher](https://www.kingfisherband.com/) [Cecilia](https://www.ceciliavacanti.com/) uses a [condenser](https://www.audio-technica.com/en-us/pro35) on her fiddle but combines it with effects pedals. This is [a bit awkward to get working](https://www.jefftk.com/p/fiddle-effects-tech), and we've been using a [Soundcraft Notepad](https://www.soundcraft.com/en-US/products/notepad-5) mini-mixer. Unfortunately we had some issues recently where adjusting the input gain would make horrible scratchy noises:

I thought I'd need to take the mixer apart to clean it, but a bit of looking online turned up suggestions to gently pull the knob off the face of the board, spray a contact treatment, and turn the spindle to work the treatment in. This seemed surprising to me: would this really distribute the cleaner where it needed to be? But I ordered some [Deoxit 5%](https://hosatech.com/products/accessories/cleaners-conditioners/caig-cleaners-conditioners/d5/) and followed the instructions on the can, and it worked perfectly!

Now I don't hear any noise at all when adjusting the gain.

I now have an almost full can of this stuff, so if you're in Boston and are having similar issues you're welcome to come borrow it.

(Even though the mixer is now working well Cecilia isn't planning to go back to it: she bought a [Voco Loco](https://www.radialeng.com/product/voco-loco), which is a box built for exactly this purpose. It should be more robust and reliable, with fewer knobs and switches that can get knocked into the wrong position.)
# Why aren't more people in AIS familiar with PDP?

Why haven't more people who work on alignment read Parallel Distributed Processing or even seem at all familiar with Rumelhart's work? This is the fundamental model of cognition and behaviour that all of modern AI is built on, the work that Hinton used for most of his insights. The model that has constantly been validated over the years by neural network performance and capabilities, but most seem incredibly unfamiliar or disinterested in it. Am I missing something? Is there some grounded reason why they think PDP and connectionism will fail at a certain point? It seems this should be required reading for anyone wanting to get into alignment research.
# What did you learn from leaked documents?

As rationalists we seek to understand the world, but it's made harder by political bias and different agendas. 

Leaked documents, represent a kind of ground truth, showing how the world really works. Telling us what's for sale, what the real agendas are, how powerful spies are, and how coordinated governments are. They are almost the opposite to conspiracy theories, as they present observations that can prune conspiracy theories.

But there are too many documents to read, so let's compare notes. What surprised you and caused you to update your view of the world?

Update: [related conversation on TheMotte.org](https://www.themotte.org/post/751/what-did-you-learn-from-leaked)
# Plum Cooking Temperature

After having fun with cooking/drying plums [last year](https://www.jefftk.com/p/making-prunes), I decided to have another go this year. This time I compared two different ways of cooking whole plums: 140F for ~1.5d vs 350F for ~1.5hr.

I was curious whether the higher temperature was better or worse. Maybe interesting flavors are lost with higher temperatures, or maybe they're lost with longer time? Maybe these approaches differ in how they convert starch to sugar?

Here's a picture of the 350F plums as I was removing them from their tray:

[![](https://www.jefftk.com/roasted-plums-tray-350f.jpg)](https://www.jefftk.com/roasted-plums-tray-350f-big.jpg)

After taking them out I manually removed the pits, and I made sure to scrape the pan with a spatula to get the tasty juice.

Here are [ready-for-rebagging](https://www.jefftk.com/p/no-more-freezer-pucks) pictures of both, 350F on the left and 140F on the right:

[![](https://www.jefftk.com/roasted-plums-350f-vs-140f.jpg)](https://www.jefftk.com/roasted-plums-350f-vs-140f-big.jpg)

I did some testing, where people didn't know which was which:

[![](https://www.jefftk.com/plum-sauce-comparison.jpg)](https://www.jefftk.com/plum-sauce-comparison-big.jpg)

Overall everyone liked the 350F better: richer plum flavor. One taster thought the 140F had a bit of an "off" flavor. The 350F was less sweet, enough so that for many dishes you wouldn't want to use it straight, but adding sugar to taste later isn't hard.

(The 350F is also way easier logistically—tying up the oven for over a day isn't great housemate behavior.)
# [Linkpost] Large language models converge toward human-like concept organization

*This is a linkpost for* [*https://arxiv.org/abs/2308.15047*](https://arxiv.org/abs/2308.15047)*. *

> Large language models show human-like performance in knowledge extraction, reasoning and dialogue, but it remains controversial whether this performance is best explained by memorization and pattern matching, or whether it reflects human-like inferential semantics and world knowledge. Knowledge bases such as WikiData provide large-scale, high-quality representations of inferential semantics and world knowledge. We show that large language models learn to organize concepts in ways that are strikingly similar to how concepts are organized in such knowledge bases. Knowledge bases model collective, institutional knowledge, and large language models seem to induce such knowledge from raw text. We show that bigger and better models exhibit more human-like concept organization, across four families of language models and three knowledge graph embeddings.

> We present a series of experiments with four families of LLMs (21 models), as well as three knowledge graph embedding algorithms. Using three different methods, we compare the vector spaces of the LLMs to the vector spaces induced by the graph embedding algorithms. (This amounts to a total of 220 experiments.) We find that the vector spaces of LLMs within each family become increasingly structurally similar to those of knowledge graph embeddings. This shows that LLMs partially converge on human-like concept organization, facilitating inferential semantics \[19\].3 The sample efficiency of this convergence seem to depend on a number of factors, including polysemy and semantic category. Our findings have important implications. They vindicate the conjecture in \[19\] that LLMs exhibit inferential semantics, settling the research question presented in \[16\], cited above. This means that LLMs partially converge toward human-like concept representations and, thus, come to partially ‘understand language’, in a non-trivial sense. We speculate that the human-like conceptual organization is also what facilitates out-of-distribution inferences in LLMs.
# Series of absurd upgrades in nature's great search

So you want to find that special thing that replicates best and lasts longest? Just vibrate a bunch of molecules a long time! 

You might reasonably assume that molecules wouldn't practically ever randomly assemble themselves into anything worth looking at. It happens a bit differently instead:


1.  Eventually, "solar systems" will form. Each one has an independent star (energy source) with several independent planets (experiment boxes) that vary in their temperature, mass, composition, etc dramatically. (Instead of just having a big homogenous soup throughout the universe with much less search power)
  - these planets will conveniently have "continents" sometimes to run more independent experiments on each one
2. A lipid layer boundary innovation thing leads to trillions of independent chemical experiments on some planets (instead of a big soup within each planet)
3. A general protein factory is discovered. Critically, it uses a simple and concise and robust physical encoding scheme. This allows good bubbles from step 2 to copy quickly and exactly, and clear out weak bubbles faster.
4. Very large multicellulared organisms make several search improvements: creatures can do sexual selection on each other instead of just getting random mutations; big smart fast predators can find & kill weak creatures quite effectively; parasites find weaknesses early & often. This all clears out weak replicators and leaves more room for the good ones.
5. Some creatures develop language and culture, which isn't  generally much use, except that it one day allows them to design new replicators from scratch.

So you really can just vibrate sand and get out all kinds of things!

A bit misleading to talk about nature's great search as natural selection via mutations+generations. It's actually much weaker (there is no algorithm or optimizer at all, only one long aimless physics sim) or uh much stronger (a five-level hierarchy having branch factor ~ 1 billion of experiments of different kinds).
# An embedding decoder model, trained with a different objective on a different dataset, can decode another model's embeddings surprisingly accurately

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2d3f13d59f8581cc8c3da9aaee3d269e77a122facb12b298.png)

(via [twitter](https://twitter.com/thesephist/status/1698095748221370575))

Seems pretty relevant to the [Natural Categories](https://www.lesswrong.com/posts/BdfQMrtuL8wNfpfnF/natural-categories-update) hypothesis.

P.S.
====

My current favorite story for "how we solve alignment" is

1.   Solve the natural categories hypothesis
2.  Add [corrigibility](https://www.lesswrong.com/posts/v3jocJRScqkBGtwvf/corrigibility-much-more-detail-than-anyone-wants-to-read)
3.  Combine these to build an AI that "does what I meant, not what I said"
4.  Distribute the code/a foundation model for such an AI as widely as possible so it becomes the default whenever anyone is building a AI
5.  Build some kind of "coalition of the willing" to make sure that human-compatible AI always has big margin of advantage in terms of computation
# Units of Exchange

Units of Exchange is the first class in the CFAR Handbook. 

In this meetup, we'll look at the tradeoffs we make in our lives, and then examine how we can make better tradeoffs.

Suggested reading (optional): [Units of Exchange](https://www.lesswrong.com/s/KAv8z6oJCTxjR8vdR/p/cvDtmPNCyrkpg4d4F)

The plan:

2:45: Doors open, snacks are brought forth  
3:00: Introductions and open conversation  
3:15: The main event  
4:45ish: Main event finishes; resume open conversation  
6:00: Meetup ends.

Note: I've never worked for CFAR or attended their classes; I'm running purely off the written handbook. If you have and you'd like to attend, I'd be delighted to have you around to course correct me!
# Open Thread – Autumn 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Strongest real-world examples supporting AI risk claims?

\[Manually cross-posted to EAForum [here](https://forum.effectivealtruism.org/posts/GDdvdhbGfCehnoJzY/strongest-real-world-examples-supporting-ai-risk-claims)\]

There are some great collections of examples of things like [specification gaming](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml?urp=gmail_link&gxids=7628), [goal misgeneralization](https://docs.google.com/spreadsheets/d/e/2PACX-1vTo3RkXUAigb25nP7gjpcHriR6XdzA_L5loOcVFj_u7cRAZghWrYKH2L2nU4TA_Vr9KzBX5Bjpz9G_l/pubhtml), and [AI improving AI](https://ai-improving-ai.safe.ai/). But almost all of the examples are from demos/toy environments, rather than systems which were actually deployed in the world.

There are also some [databases](https://incidentdatabase.ai/) of AI incidents which include lots of real-world examples, but the examples aren't related to failures in a way that makes it easy to map them onto AI risk claims. (Probably most of them don't in any case, but I'd guess some do.)

I think collecting real-world examples (particularly in a nuanced way without claiming too much of the examples) could be pretty valuable:

*   I think it's good practice to have a transparent overview of the current state of evidence
*   For many people I think real-world examples will be most convincing
*   I expect there to be more and more real-world examples, so starting to collect them now seems good

**What are the strongest real-world examples of AI systems doing things which might scale to AI risk claims?**

I'm particularly interested in whether there are any good real-world examples of:

*   Goal misgeneralization
*   Deceptive alignment (answer: [no](https://www.lesswrong.com/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1), but yes to simple deception?)
*   Specification gaming
*   Power-seeking
*   Self-preservation
*   Self-improvement

This feeds into a project I'm working on with AI Impacts, collecting empirical evidence on various AI risk claims. There's a work-in-progress table [here](https://docs.google.com/spreadsheets/d/15pW8_pwbznnbQBu0Ogpk9qtqrSPKOoIjzbwe58GM4Ls/edit#gid=0) with the main things I'm tracking so far - additions and comments very welcome.
# Conversation about paradigms, intellectual progress, social consensus, and AI

I have a number of thoughts here that I haven't gotten around to writing up, and sharing via a public conversation seems like a decent way to try.  
  
A few opening thoughts:

*   People talk about the field of AI Alignment being "pre-paradigmatic". I don't have a sense that people have shared precise models behind this, or at least I don't know what other people mean when they say it. Most people don't seem to have read Kuhn for example..
*   I used to think the goal was to "become paradigmatic". I now think a better goal is to become the kind of field that can go through cycles of paradigm creation, paradigm crisis, and then recreation of a new paradigm.
*   I think there are ways to increase our ability to do that, and I'd like to have people thinking more about them.

Happy to launch off in any direction that seems interesting or productive to you.
# My First Post

Hello all!

I'm Jaivardhan Nawani, a 12-year-old enthusiast on rationality and how using it can improve our day-to-day lives.

Recently, I discovered this forum from a few friends who introduced me to many useful principles, and I thought to come up with an idea to make rationality easier to implement for first-timers.

So here's a basic theorem that I made upon this concept. I've come to LessWrong looking for feedback on it. And that's mostly it.

Here's the link to my theorem: https://docs.google.com/document/d/1Y2jvSR6RNFp_l-D-wX-KjHDrNUyhfdcM/edit?usp=sharing&ouid=100577114717816710937&rtpof=true&sd=true

Have a good day and with regards;

Jaivardhan Nawani.
# Video essay: How Will We Know When AI is Conscious?


I recently came across this engaging video essay by [exurb1a](https://www.youtube.com/@Exurb1a) and felt it would be of interest to this community. Contrary to its title, it avoids the common pitfall of asserting that only conscious AI can be dangerous, and it effectively addresses several of the main topics often discussed on this forum.

If you've ever wanted a lighthearted resource to share with friends less familiar with these concepts, this might be a good starting point.
# Weekly Incidence vs Cumulative Infections

Imagine you have a goal of identifying a novel disease by the time some small fraction of the population has been infected. Many of the signs you might use to detect something unusual, however, such as doctor visits or shedding into wastewater, will depend on the number of people *currently* infected. How do these relate?

Bottom line: if we limit our consideration to the time before anyone has noticed something unusual, where people aren't changing their behavior to avoid the disease, the vast majority of people are still susceptible, and spread is likely approximately exponential, then:

incidence = cumulative infections ln (2) doubling time

Let's derive this! We'll call "cumulative infections" c(t), and "[doubling time](https://en.m.wikipedia.org/wiki/Doubling_time)" Td. So here's cumulative infections at time t:

c(t) = 2 t Td

The math will be easier with natural exponents, so let's define k = ln (2) Td and switch our base:

e kt

Let's call "incidence" i(t), which will be the derivative of c(t):

i(t) = d dt c(t) = d dt e kt = k e kt

And so:

i(t) c(t) = k e kt e kt = k = ln (2) Td

Which means: i(t) = c(t) ln (2) Td

What does this look like? Here's a chart of weekly incidence at the time when cumulative incidence reaches 1%:

[![](https://www.jefftk.com/weekly-incidence-when-1pct.png)](https://www.jefftk.com/weekly-incidence-when-1pct-big.png)

For example, if it's doubling weekly then when 1% of people have ever been infected 0.69% of people became infected in the last seven days, representing 69% of people who have ever been infected. If it's doubling every three weeks, then when 1% of people have ever been infected 0.23% of people became infected this week, 23% of cumulative infections.

Is this really right, though? Let's check our work with a bit of very simple simulation:

def simulate(doubling\_period\_weeks):
  cumulative\_infection\_threshold = 0.01
  initial\_weekly\_incidence = 0.000000001
  cumulative_infections = 0
  current\_weekly\_incidence = 0
  week = 0
  while cumulative_infections < \
        cumulative\_infection\_threshold:
    week += 1
    current\_weekly\_incidence = \
        initial\_weekly\_incidence * 2**(
          day/doubling\_period\_weeks)
    cumulative_infections += \
        current\_weekly\_incidence

  return current\_weekly\_incidence

for f in range(50, 500):
  doubling\_period\_weeks = f / 100
  print(doubling\_period\_weeks,
        simulate(doubling\_period\_weeks))

This looks like:

[![](https://www.jefftk.com/weekly-incidence-when-1pct-simulated.png)](https://www.jefftk.com/weekly-incidence-when-1pct-simulated-big.png)

The simulated line is jagged, especially for short doubling periods, but that's not especially meaningful: it comes from running the calculation a week at a time and how some weeks will be just above or just below the (arbitrary) 1% goal.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid04r7XiT3AK6VftKPqj4M4u6fbLVNNExmqAA1DuTYnfMkQLNFSvDAUZNNMwYsHkq5ul), [mastodon](https://mastodon.mit.edu/@jefftk/111021490637605604)*
# How did you make your way back from meta?

I've noticed in myself a strong preference for focusing on the meta level. It's most visible in fields dear to me, like writing. For example, I'll spend much more time reading up on rhetoric or tracking down rare 1980's books about writing techniques than practicing writing essays or stories.

I don't like this because my ultimate goal in studying the meta level is to get better at the object level. At the same time, I am sometimes rewarded for going so meta. I've gotten a lot of respect at work paired with feedback that I provide excellent feedback and perspectives.

There don't seem to be immediately painful effects of this state. I have a family and a job and my life seems in order overall. But there's a hunger for a) putting theory to the test and seeing results (ie. making meta pay rent), and b) learning from direct experiences & sharing those experiences with others. Meta is a lonely place to be.

I don't think I'm the only one in this position. I found these two posts with just a few seconds of searching (I'm sure there's more):
https://www.lesswrong.com/posts/g2AKPEzFdQitmpTDu/meta-addiction
https://www.lesswrong.com/posts/RnP5bR767NcxebYHd/conjecture-on-addiction-to-meta-level-solutions


The last few days, I've been catching myself starting on a meta-deepening activity and consciously switching to an object-level task. It's been rewarding so far, so I believe that in a few weeks, my habits will shift toward where I want to be.

But I'm curious: has anyone else experience something similar? How did it go for you? What did you do?
# Bring back the Colosseums

Men want to engage in righteous combat. They want it more than they want sex or VP titles. They fantasize about getting the casus belli to defend themselves against armed thugs that will never come, they spend billions of dollars on movies and TV about everymen in implausible circumstances where EA calculus demands they use supernatural powers for combat, they daydream about [fantastical](https://en.wikipedia.org/wiki/Naruto), [spartan](https://en.wikipedia.org/wiki/Fullmetal_Alchemist) settings where war is omnipresent and fights are personal and dramatic and intellectually interesting, and they're basically incapable of resisting the urge to glorify their nation and people's past battles, even the ones they claim to disagree with intellectually. You cannot understand much of modern culture until you've recognized that the state's blunt suppression of the male instinct for glory has caused widespread symptoms of pica that dominate our politics, media, and online interactions.

And make no mistake - our half-hearted policy of deeming all such tendencies "toxic masculinity", and refusing men the option to engage in reciprocal or consensual violence against each other, has been a bigger failure than the war on drugs. Lots of ink has been spilled on sphere of influence conspiracy theories that attempt to interpret America's foreign adventures as rational power-seeking behavior. But the real truth is that men naturally form gangs, political cliques, and military theologies that attempt to justify violence within their existing legal and moral landscapes independent of any external incentives to do so. What they really want from all this is not some policy outcome but the self-actualization that comes from fighting the enemy, and the dearth of opportunities for them to challenge their opponents' honor on the battlefield in a rights-respecting way is a much more important misandrist failing than child custody bias or divorce law or anything I've seen red pill people argue on the internet about. Men who are down especially bad will take absurd pay cuts to join [artificial and economically unmotivated criminal sects](https://en.wikipedia.org/wiki/Shedden_massacre), solely so they get the opportunity to pick mortal battles with other people who've opted into the same social systems they have. 

There is no true modern substitute for these ambitions, with all of their cultural and social significance. Yet the compassionate solution is obvious. A naturally belligerent people, like Americans, *require* a social outlet like the arena; more than that they need a supportive, dueling-appropriate culture. Thus, a modest proposal:

*   Legalize all death matches between consenting adults.
*   Allow the commercialization of duels, side bets on outcomes, and customized sporting arenas for such venues.
*   Tolerate the return of America's premodern honor system. Resist the forces of media virtue-signaling and authoritarianism that have snuffed one of our greatest traditions.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/02f99d57b4f1800a4e3f31483938add2e220b7720e597b31.png)

I am not joking.
# What is the optimal frontier for due diligence?

The title isn't quite right.  My current take is that:

*   There is a ~10k word [post](https://www.lesswrong.com/posts/Lc8r4tZ2L5txxokZ8/sharing-information-about-nonlinear-1) sharing some damaging information about someone else.
*   3-4k words are describing accusations that are disputed.
    *   However, a few have already had substantial [evidence](https://www.lesswrong.com/posts/Lc8r4tZ2L5txxokZ8/sharing-information-about-nonlinear-1?commentId=hnPderw8aGqA8dwQS) provided against them.
    *   The post author [doesn't think the accuracy of those accusations is particularly cruxy](https://www.lesswrong.com/posts/Lc8r4tZ2L5txxokZ8/sharing-information-about-nonlinear-1?commentId=hnPderw8aGqA8dwQS) for their major takeaways.
*   Those accusations generally sound pretty bad.
*   The post doesn't explain why the (disputed) accusations are salient, if they aren't cruxy for the post author's major takeaways.
*   The post author may not find those accusations cruxy, but I would be surprised if nobody else found them cruxy for deciding between "there's some bad stuff in here, but it seems possible that most of the harm was caused by some combination of large cultural differences and poor communication", and "nope nope nope".
    *   This is to say, our actions have effects beyond the EA ecosystem.  I think it's wrong to decide that, because other people may update incorrectly based on evidence you've provided, the harm that comes from those updates doesn't matter much.

The thing I'm noticing is some kind of [missing mood](https://www.econlib.org/archives/2016/01/the_invisible_t.html).  I think that there's been a failure to inhabit [the least convenient possible world](https://www.lesswrong.com/posts/neQ7eXuaXpiYw7SBy/the-least-convenient-possible-world), and the general distribution over possible outcomes, and correspondingly attempt to move to the pareto-frontier of outcomes assuming that distribution.  This comment in particular seems to be operating in the frame of "You weren't behaving the way I'd update positively on if the accusations were true".  In the world where the disputed accusations are in fact false, for the specific reasons they provided, I think it would be pretty strange to take the suggested course of action.
# AI presidents discuss AI alignment agendas

None of the presidents fully represent my (TurnTrout's) views.

TurnTrout wrote the script. Garrett Baker helped produce the video after the audio was complete. Thanks to David Udell, Ulisse Mini, Noemi Chulo, and especially Rio Popper for feedback and assistance in writing the script.
# Feature proposal: Export ACX meetups

Currently there are two places that display upcoming meetups of Astral Codex Ten:

*   [Community Section](https://www.lesswrong.com/community) of LessWrong
*   articles at ACX, such as "[Meetups Everywhere 2023: Times & Places](https://www.astralcodexten.com/p/meetups-everywhere-2023-times-and)"

Note that there are *almost 200* meetups this autumn. Sometimes a mistake happens, and meetups that are displayed at one place are missing from the other place. As a consequence, some people miss the information about an upcoming meetup.

I don't know how exactly this works today. My best guess is that someone prepares a spreadsheet, and then imports it to both places. That part seems to work nicely.

Problems happen when the information changes later. People announce new ACX meetups after the deadline. Also, meetup organizers can edit the information on LW, but the changes do not automatically propagate to ACX.

ACX runs on Substack, which is a generic blogging platform. There is no specialized support for meetups, only a WYSIWYG editor for articles.

If these assumptions are generally correct, I propose to modify the process like this.

**Step one**: After collecting information to the spreadsheet, import the data to LessWrong. Then throw the spreadsheet away, because now LessWrong becomes the *source of truth*. (Meetup organizers can edit their announcements.)

Any meetups that are announced after the deadline, must be created on LessWrong. It is possible to reuse the same process: collect them to a (new!) spreadsheet, and import it. But they can also be created manually, whatever is more convenient.

**Step two**: Make an export from LessWrong for Astral Codex Ten, in format that is most convenient for Scott to import. Probably rich text or HTML code, which he could simply copy and paste to the article, *without having to edit it further*. That means, that the exported meetups would be organized in Scott's preferred way (alphabetically, groupped by continent and city) and properly formatted.

That means, at first, Scott would create a new article, copy the data from LessWrong, and write an introduction. After changes, Scott would delete the imported part from the existing article, and copy the new export from LessWrong.

Afterwards he can compare the old export and the new export (e.g. using [this tool](https://text-compare.com/)) and announce the major changes in the next Open Thread. This part cannot be fully automated, because there are various kinds of possible changes, some of them important (new meetup, cancelled meetup, change of date and time, change of place), some unimportant (updated description, fixed a typo), therefore human judgment is needed.

As a bonus, export maps of continents, marking the places where the meetups occur. (Even if someone did this manually for ACX, they wouldn't bother updating the maps.)
# What are some good language models to experiment with?

Like if I want to experiment with a steering technique, it would be useful to have a language model that is small, capable, but not so finetuned that it becomes inflexible. (Or maybe ideally, a model which has both a finetuned and a non-finetuned variant.)

I've seen some people use GPT-2. Is that recommended? Are there any alternatives?
# High school advice

What advice does the Lesswrong community have in general for high school students who stumbled on this community?   
  
Looking for all types of responses, it's fine whether your response is focused on achieving something in the AI/effective altruism/rationality community, or whether it is focused on a more general audience.
# Focus on the Hardest Part First

Consider reading [this](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html) instead.

Here is some obvious advice.

I think a common failure mode when working on AI alignment[^1] is to not focus on the hard parts of the problem first. This is a problem when generating a research agenda, as well as when working on any specific research agenda. Given a research agenda, there are normally many problems that you know how to make progress on. But blindly working on what seems tractable is not a good idea.

Let's say we are working on a research agenda about solving problems A, B, and C. We know that if we find solutions to A, B, and C we will solve alignment. However, if we can't solve even one subproblem, the agenda would be doomed. If C seems like a very hard problem, that you are not sure you can solve, it would be a bad idea to flinch away from C and work on problem A instead, when A seems so much more manageable.

If solving A takes a lot of time and effort, all of that time and effort would be wasted, if you can't solve C in the end. It's especially worrisome when A has tight fightback loops, such that you constantly feel like you are making progress. Or when it is just generally fun to work on A. 

Of course, it can make sense to work on A first if you expect this to help you solve C, or at least give you more information on its tractability. The general version of this is illustrated by considering that you have a large list of problems that you need to solve. In this case, focusing on problems that will provide you with information that will be helpful for solving many of the other problems *can* be very useful. But even then you should not lose sight of the hard problems that might block you down the road.

The takeaway is that these two things are very different:
- Solving A as an instrumental subgoal in order to make progress on C, when C is a potential blocker.
- Avoiding C, because it seems hard, and instead working on A because it seems tractable.

[^1]: Though I expect this to be a general problem that comes up all over the place.
# A Bat and Ball made me Sad

'A bat and a ball cost $110 in total.

The bat costs $100 more than the ball.

How much does the ball cost?

**The answer is $5.**

**Please enter the number 5 in the blank below.**

 $_____'  
  
That question is from one of 59 new studies (*N* = 72,310) which focus primarily on the “bat and ball problem," found in [The formation and revision of intuitions - ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0010027723000148?) by Andrew Meyer and Shane Frederic.  
  
23% of respondents failed to give the correct answer.    
  
I don't have anything insightful to say but wanted to share because it seems worth knowing.    
  
Additionally, I'm not exactly sure how (and how much) I should update about this.   
  
(Edit: I initially posted the wrong question)