Harley-ml/lesswrong huggingface dataset:
# Cleaning a Spoon is Complex

A friend of mine recently shared this:

[![](https://www.jefftk.com/max-temkin-spoon-poster.jpg)](https://www.jefftk.com/max-temkin-spoon-poster-big.jpg)

I've seen it go by dozens of times since Max Tempkin \[1\] made it [back in 2011](https://web.archive.org/web/20111010045930/http://www.maxistentialism.com/prints), and each time it grated on me: why count the complexity of creating the disposable spoon, but not the complexity of washing it? Instead of phrasing my disagreement as a comment, though, I decided to make a new version:

[![](https://www.jefftk.com/spoon-cleaning-complexity.png)](https://www.jefftk.com/spoon-cleaning-complexity-big.png)

You know the thing where you see something on Facebook, start reading it or composing a reply, and then you lose it? Yeah. I can't remember which of my friends shared this, and Facebook's limited-by-design searching didn't turn it up either. So, being unable to post my counter-meme in the appropriate place, I'll just leave it here until the next time I come across the original.

  
\[1\] Also known for co-designing [Cards Against Humanity](https://en.wikipedia.org/wiki/Cards_Against_Humanity), from which he [resigned in disgrace](https://web.archive.org/web/20200805005502/https://www.cardsagainsthumanity.com/statement/) in June 2020.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0fkTqaBHLVqRfTakRefaiGTmCTUwDdp9vUdRNoBc5yoaNnSFtxRKpMpuuvM5Stb6pl)*
# [Crosspost] AlphaTensor, Taste, and the Scalability of AI

My big takeaway from the AlphaTensor paper is that DeepMind have extraordinary *taste*: they're able to identify problems that their approach to AI can tackle well, today; and they can figure out how to regiment those problems in such a way as make the AI tackle it:

> Their approach is a variant of the deep reinforcement-learning-guided Monte Carlo tree search that they have applied so successfully to playing Chess and Go. What they have done, very effectively, is to design a game with the objective of finding the most efficient tensor multiplication algorithm for a matrix of some dimension.

On the presupposition that we don't get AGI, much more AI research will look a lot like this. Find a good question, figure out how to regiment the question into the constraints imposed by the model's architecture, and apply the model to answer the question.

But those skills don't look like the sort of thing that benefit from scaling compute:

> This ability – being able to reorganise a question in the form of a model-appropriate game – doesn’t look nearly as susceptible to Moore’s Law-style exponential speed-ups. Researchers’ insights and abilities – in other terms, researcher productivity – don’t scale exponentially. It takes time and energy and the serendipitousness of a well-functioning research lab to cultivate them. Scaling compute down to an effective cost of zero doesn’t help if we’re not using these models to attack the right issues in the right way.

So the bottleneck won't be compute! The bottleneck will be the sort of excellent *taste* that DeepMind keep displaying.
# AMC's animated series "Pantheon" is relevant to our interests

Sorry for the vague title. I'm avoiding spoilers for those who care a lot about that sort of thing.

SPOILERS FOLLOW

* * *

The series illustrates a fast take-off scenario. It tells the story of a fast-take-off scenario of "uploaded intelligence". Of course the usual fast take-off scenario this community is concerned about is artificial intelligence, not uploaded intelligence (i.e., digitized human intelligence as in Robin Hanson's *Age of Em*). But the series portrays the dangers of uploaded intelligence in a remarkably similar way to the concerns many people have about a fast AI take-off.

It feels very embedded in the state of the current world in 2022 in many ways; referencing the pandemic in ways most shows don't dare to; set in perhaps an alternative history where we weren't on the cusp of AGI, but where a Steve Jobs-like character had almost, but not quite, cracked the brain-uploading problem 20 years ago.

There are plenty of plot devices that don't feel realistic, but the interaction between technology and geopolitics, the exponential growth when AI take-off happens, the pivotal nature of very specific events guiding the direction of the take-off at the time it occurs--all that tracks, for me.

Worth watching as entertainment for yourself, and worth popularizing as a way to prime people out there on the dangers of a fast AI take-off in the near future.

That is all I have to say! Just putting the alert out there.
# Does biology reliably find the global maximum, or at least get close?

Jacob Cannell has claimed that biological systems at least get within 1 OOM of not a local, but global maximum in abilities.

His comment about biology nearing various limits are reproduced here:

> The paper you linked seems quite old and out of date. The modern view is that the inverted retina, if anything, is a superior design vs the everted retina, but the tradeoffs are complex.

> This is all unfortunately caught up in some silly historical "evolution vs creationism" debate, where the inverted retina was key evidence for imperfect design and thus inefficiency of evolution. But we now know that evolution reliably finds pareto optimal designs:

> biological cells operate close to the critical Landauer Limit, and thus are pareto-optimal practical nanobots.

> eyes operate at optical and quantum limits, down to single photon detection.

> the brain operates near various physical limits, and is probably also near pareto-optimal in its design space.

Link to comment here: 

https://www.lesswrong.com/posts/GCRm9ysNYuGF9mhKd/?commentId=aGq36saoWgwposRHy

I am confused about the Landauer limit for biological cells other than nerve cells, as it only applies to computation, but I want to ask, is this notion actually true?

And if this view was true, what would the implications be for technology?
# Misalignment Harms Can Be Caused by Low Intelligence Systems

Most clinical trials are very simple statistical models. There are not trillions of parameters in a complex neural network architecture. Yet they are still the most effective tools we have at validating and optimising health and recovery in human beings. Very simple statistical tools can lead to large effects and, if misaligned, very large harms.

When private companies use A/B tests on human beings, they are usually explicitly optimising for objectives which are not aligned with human wellbeing - engagement, click-through rate, company profits. The harm that this can then cause depends on the nature of the interaction surface between the technology and the human, and the amount of exposure.

In configuration space we can consider the total effect of a company on a human as $\Delta H|_C=\int{\frac{\partial H}{\partial C}}|_t dt$ where $\frac{\partial H}{\partial C}|_t$ is the 'interaction bandwidth' between human and company across the human's markov blanket at time t. As an approximation, we can say the magnitude of the overall interaction is going to be roughly proportional to the magnitude of the average interaction bandwidth multiplied by the cumulative interaction time:

$|\Delta H|_C \approx\langle\frac{\partial H}{\partial C}\rangle\cdot T$

In modern social media, $T$ is obviously high relative to many other interventions, but what about $\langle\frac{\partial H}{\partial C}\rangle$? What makes the interaction bandwidth high or low? Is this determined by the the complexity of the recommendation algorithm you use? Is this determined by the potency of the content available to recommend?

If there was a piece of content so convincing that anyone who read it instantly decided to sell all of their possessions, give the proceeds to MegaCorp and spend the rest of their life giving any available money to MegaCorp, then all the MegaCorp Recommendation Algorithm™ would have to do to maximise profits would be to show this content. It would not require a complex statistical model. Now this content may not exist, but participants in the platform are strongly incentivised to try and create it as conforming with the utility of the platform leads to broad reach again using a simple recommendation system.

Now, once you do have a highly intelligent recommendation system optimising on humans with knowledge of the human's present and past states, the interaction between recommendation system and human is no longer ergodic and there can be more powerful interactions constructed over time with a tight feedback loop on the human. My point is, that this is *not necessary* for severe effects and severe harms to be caused by misaligned systems.
# Prettified AI Safety Game Cards

Donald Hobson's [AI Safety Card Game](https://www.lesswrong.com/posts/Nex8EgEJPsn7dvoQB/the-ai-safety-game-updated) already has simple cards you can print out on cardstock and cut out, but I wanted a nicer print-on-demand version with illustrated card backs.[^vnjwx1uov2k] [You can order it here](https://www.thegamecrafter.com/games/ai-safety-card-game). Major thanks to Mati Roy for helping make this happen, along with Jim Babcock, and Daniel Recinto.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/96d5f0c29596f13d51e167e38176cb5cc521d14345fbd1d4.jpg)

[^vnjwx1uov2k]: Card-back illustrations are AI-generated. We did this around December-January, when the state of the art for AI-generated illustrations was much worse! I would be excited for someone to re-do it now. There's a lot of room for improvement; for example, illustrated card fronts rather than just card backs.
# That one apocalyptic nuclear famine paper is bunk

I've seen a few commentors and bloggers cite [this study](https://www.nature.com/articles/s43016-022-00573-0) from Nature Food to explain why they think nuclear war may lead to the collapse of civilization in event of a full nuclear exchange between the US and Russia. I read it and updated towards Nature Food being a publication with ~~bad~~ terrible editorial standards more than anything else. 

### The Apocalypse is Bad at Math

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e6232712057f9f8b7422910df2a556945530b3fd0c3df7fc.png)

Wow, 5 Billion people dead. That's pretty bad. It actually projects 99% population reduction in China, the US, and Russia due to a 90% reduction in global farm yields. But let's break down those numbers.  Firstly, how did they even get those crop yield numbers? Well, it turns out they assumed a 10 degrees C decrease in temperatures in the northern hemisphere, and then assumed no change in the crop selection and area under cultivation.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b1da8c0708360bca4d9fca7a3cc94be700546d1df2a51016.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2496a85b7c9d0796e5c0ca74665aacd9dd85a1c78b6e8fa4.png)

This isn't just bad statistics. It's intentionally misleading. Of course, if temperatures drop 10 degrees and people do literally nothing in response crop yields will decrease massively. But that's not how human beings work. They will increase land under cultivation in the tropics, clear-cut the Amazon, and do whatever it takes to get food production going again. Obviously, they will switch to more cold-resistant crops rather than lose 100% of their harvest to frost. These assumptions are terrible. Garbage in, garbage out.

This is a bad paper. Other scientists in the field know it's a bad paper, which is why almost no one has cited it. It's a scaremongering piece designed to push a denuclearization agenda and is used as clickbait by journalists. Its bad assumptions make it basically useless. Please don't take any apocalyptic nuclear starvation scenarios based on it seriously. I am utterly astounded that a journal like Nature let this one through. I will definitely try to publish in Nature Food in the future.
# Alignment 201 curriculum

This curriculum, a follow-up to the [Alignment Fundamentals curriculum](https://www.agisafetyfundamentals.com/ai-alignment-curriculum) (the ‘[101](https://en.wikipedia.org/wiki/101_(slang))’ to this 201 curriculum), aims to give participants enough knowledge about alignment to understand the frontier of current research discussions. It assumes that participants have read through the Alignment Fundamentals curriculum, taken a course on deep learning, and taken a course on reinforcement learning (or have an equivalent level of knowledge).

Although these are the basic prerequisites, we expect that most people who intend to work on alignment should only read through the full curriculum after they have significantly more ML experience than listed above, since upskilling via their own ML engineering or research projects should generally be a higher priority for early-career alignment researchers.

When reading this curriculum, it’s worth remembering that the field of alignment aims to shape the goals of systems that don’t yet exist; and so alignment research is often more speculative than research in other fields. You shouldn’t assume that there’s a consensus about the usefulness of any given research direction; instead, it’s often worth developing your own views about whether techniques discussed in this curriculum might plausibly scale up to help align AGI.

The curriculum was compiled, and is maintained, by [Richard Ngo](http://richardcngo.com/). For now, it’s primarily intended to be read independently; once we’ve run a small pilot program, we’ll likely extend it to a discussion-based course. 

**Curriculum overview**

Week 1: Further understanding the problem  
Week 2: Decomposing tasks for better supervision  
Week 3: Preventing misgeneralization  
Week 4: Interpretability  
Week 5: Reasoning about Reasoning  
Weeks 6 & 7 (Track 1): Eliciting Latent Knowledge  
Weeks 6 & 7 (Track 2): Agent Foundations  
Weeks 6 & 7 (Track 3): Science of Deep Learning  
Weeks 8 & 9: Literature Review or Project Proposal

[**See the full curriculum here**](https://www.agisafetyfundamentals.com/alignment-201-curriculum). Note that the curriculum is still under revision, and feedback is very welcome!
# Another problem with AI confinement: ordinary CPUs can work as radio transmitters

Excerpts:

> Computer tech expert William Entriken has created a C library that can make a computer emit radio waves even if the device doesn't include any radio transmission hardware.
> 
> His program, called System Bus Radio, is written in C and relies on code that makes the computer's CPU emit electromagnetic radiation at certain frequencies.
> 
> The radiation is strong enough to escape the computer's shielding, and with the proper timing at which data is processed in the CPU, it can be forged into radio waves that carry off information away from the target's computer.
> 
> During tests, Mr. Entriken says he managed to transmit a version of the "Mary Had a Little Lamb" song over a distance of up to two meters and one meter if a drywall separated the computer and the receiving antenna.
> 
> You can get System Bus Radio from [GitHub](https://github.com/fulldecent/system-bus-radio), test it on your own hardware, and add your results to Mr. Entriken's findings.

Worth reminding that this escape method was invented by a human-level intelligence. A Bayesian superintelligence may be able to devise much more cunning methods.
# Metaculus Launches the 'Forecasting Our World In Data' Project to Probe the Long-Term Future

### *With support from the FTX Future Fund, the project features a public forecasting tournament and a Pro Forecaster initiative delivering 1, 3, 10, 30, and 100-year predictions*

![](https://metaculus-media.s3.amazonaws.com/user_uploaded/owid-final-poster.jpeg)

[Metaculus](https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/) has launched [*Forecasting Our World In Data*](https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/), a tournament that will deliver predictions on technological advancement, global development, and social progress on time horizons ranging from one to 100 years, with close collaboration from the [Our World In Data](https://ourworldindata.org/) (OWID) team.

OWID is one of the world's largest open online repositories of data and is a valuable source of information about both humanity's progress and challenges. Metaculus will collect forecasts and analyses on thirty OWID metrics — including CO2 emissions, supercomputer performance, nuclear stockpiles, [and more](https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/) — to generate likelihoods of specific outcomes in humanity's future. Aggregating crowd forecasts and weighting them by forecasters' past performance leads to [greater accuracy](https://pubmed.ncbi.nlm.nih.gov/24659192/) than relying on a single model or subject matter expert alone.

![](https://metaculus-media.s3.amazonaws.com/user_uploaded/prize-structure_bLNrjPM.jpeg)

The *Forecasting Our World In Data* tournament will award a prize pool of $20,000 to incentivize accurate forecasting and insightful written comments by forecasters. For each OWID metric, participants are invited to predict outcomes at 1, 3, 10, 30, and 100 years. Prizes will be awarded for accurate performance on 1 and 3-year outcomes and cogent analysis on 10, 30, and 100-year outcomes. Questions are staggered, with 10 questions opening today and 10 additional questions opening October 19th and 26th. [Metaculus Pro Forecasters](https://www.metaculus.com/help/faq/#what-are-pros) will predict each metric alongside the crowd.

A grant from the [Future Fund](https://ftxfuturefund.org/) makes this project possible, and is greatly appreciated. Our thanks also to the forecasting community for contributing to a greater understanding of humanity's near- and long-term future. The first set of questions is now open for public forecasting [here](https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/).
# Best resource to go from "typical smart tech-savvy person" to "person who gets AGI risk urgency"?

Background
==========

Most tech-savvy people today see AI as a cool tech trend like the internet / smartphones / crypto / virtual reality. The claim "AGI is pretty likely to kill you and your loved ones in your lifetime" has never been a belief they take seriously, or think that many other people take seriously. They also don't intuitively perceive AGI as a risk that's relevant in their own lifetime; only as a low-probability long-term risk like an asteroid impact.

The Needed Resource
===================

Something on the internet they can read/watch to try to guide such a person to the point where they realize maybe AI existential risk is a real urgent thing, at least as much so as climate change. Someone who gets that they might be ringing in the new decade as a nonhuman configuration of atoms.

Benefit of Having This
======================

The benefit is that a significant fraction of the people who get AGI risk urgency will be motivated to do something helpful to the cause, much more so than the fraction of people who don't get AGI risk urgency.

Some Candidate Resources
========================

Two off the top of my head:

[\[WaitButWhy\] The AI Revolution: Our Immortality or Extinction](https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html)

[\[LessWrong\] AGI Ruin: A List of Lethalities](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)

The WaitButWhy post does a great job of being generally accessible, while List of Lethalities packs in more of the arguments why the danger level seems high. It seems like there's a gap for a resource that's more suitable to the task of making a normal person realize this is a real and urgent thing, despite the social Overton Window not yet giving them that message.

If there's no good resource, I might be up for working on something myself. Maybe a good format would be similar to that of a technical documentation site, where it's kind of a linear walkthrough but also lets you skip around to sections you're interested in.
# Halifax, NS – Monthly Rationalist, EA, and ACX Meetup Kick-Off

In the interest of growing the LW/EA/ACX community in Halifax, we will be hosting meetups on the last Saturday of each month. The format is friendly, unstructured discussion (future iterations could emphasise particular topics). Old hands and newcomers are welcome!

Location: Seven Bays Cafe (2017 Gottingen Street) – [87PRMC29+9C](https://plus.codes/87PRMC29+9C)

Join us at Seven Bayes

Contact: conorbarnes93@gmail.com
# They gave LLMs access to physics simulators

Over at Google, large language models have been plugged into physics simulators to help them share a world model with their human interlocutors, resulting in big performance gains. They call it Mind's Eye.  
  
This is how the authors describe the work:

> Correct and complete understanding of properties and interactions in the physical world is not only essential to achieve human-level reasoning (Lake et al., 2017), but also fundamental to build a general-purpose embodied intelligence (Huang et al., 2022). In this work, we investigate to what extent current LMs understand the basic rules and principles of the physical world, and describe how to ground their reasoning with the aid of simulation. Our contributions are three-fold:
> 
>  • We propose a new multi-task physics alignment dataset, UTOPIA, whose aim is to benchmark how well current LMs can understand and reason over some basic laws of physics (§2). The dataset contains 39 sub-tasks covering six common scenes that involve understanding basic principles of physics (e.g., conservation of momentum in elastic collisions), and all the ground-truth answers are automatically generated by a physics engine. We find that current large-scale LMs are still quite limited on many basic physics-related questions (24% accuracy of GPT-3 175B in zero-shot, and 38.2% in few-shot).
> 
>  • We explore a paradigm that adds physics simulation to the LM reasoning pipeline (§3) to make the reasoning grounded within the physical world. Specifically, we first use a model to transform the given text-form question into rendering code, and then run the corresponding simulation on a physics engine (i.e., MuJoCo (Todorov et al., 2012)). Finally we append the simulation results to the input prompts of LMs during inference. Our method can serve as a plug-and-play framework that works with any LM and requires neither handcrafted prompts nor costly fine-tuning.
> 
>  • We systematically evaluate the performance of popular LMs in different sizes on UTOPIA before and after augmentation by Mind’s Eye, and compare the augmented performance with many existing approaches (§4.2). We find Mind’s Eye outperforms other methods by a large margin in both zero-shot and few-shot settings. More importantly, Mind’s Eye is also effective for small LMs, and the performance with small LMs can be on par or even outperform that of 100× larger vanilla LMs.

This seems like a direct step down the [CAIS](https://www.lesswrong.com/posts/SbadvzWbufzX9iWJf/they-gave-llms-access-to-physics-simulators?commentId=nibBmT824AJMjNogG) path of development.
# Vertical Flutes

The standard (" [transverse](https://en.wikipedia.org/wiki/Transverse_flute)") flute has pretty terrible ergonomics: your arms are way out to the side. Most woodwinds use a much more natural playing position where the arms are comfortably in front of you, including some (" [end-blown](https://en.wikipedia.org/wiki/End-blown_flute)") flutes. In the case of the flute, however, this gives a pretty different sound. Could we give up on the simple cylindrical design of the flute to get some thing that sounds like a transverse flute and plays like a transverse flute but is more ergonomic?

Yes! By putting a bend in the head joint it's possible to do just that. This is a common configuration for bass flutes and lower:

[![](https://www.jefftk.com/jupiter-bass-flute.jpg)](https://www.jefftk.com/jupiter-bass-flute-big.jpg)  
*[Jupiter JBF1100E](http://jupitermusic.com/international/products/woodwinds/flutes/bass-flutes/jbf1100e/) bass flute*

[![](https://www.jefftk.com/pearl-contrabass-flute.png)](https://www.jefftk.com/pearl-contrabass-flute-big.png)  
*[Pearl PFC-905](https://www.pearlflute.com/products/contrabass-flute) contrabass flute*

You can get a replacement head joint for a standard concert flute, though it's pricy:

[![](https://www.jefftk.com/flute-lab-vertical-flute.jpg)](https://www.jefftk.com/flute-lab-vertical-flute-big.jpg)  
*[Flute Lab Vertical Headjoint](https://www.flutespecialists.com/product/flute-lab-vertical-headjoint/)*

I don't know why this isn't more common? Do people not like that it looks unusual? Katherine Saenger looked into this some ([pdf](https://www.nyfluteclub.org/uploads/2020-2021/FluteFair-2021/FluteFair2021-angled-headjoints2-low.pdf)) and claims that the main options for concert flute today don't sound great but better-sounding ones are possible. Has anyone tried these?
# Audition to perform in Bay Secular Solstice

We’re looking to get new people involved with Secular Solstice this year! We need orators to give speeches, and singers and instrumentalists of all kinds to lead singalongs.

Solstice will be on December 17th in SF or Oakland, and there will be a dress rehearsal in the week prior. Those are the only two times you need to be available; most of the preparation can be done on your own, so performing doesn't necessarily need to be a significant time commitment. (You don't even necessarily need to live in the Bay!)

Audition by recording a video and submitting it through [this form](https://forms.gle/xez4SjguTm7VF5QSA) 🙂
# Is GitHub Copilot in legal trouble?

This is basically a crosspost for [https://githubcopilotinvestigation.com/.](https://githubcopilotinvestigation.com/.) I noticed that some folks in California are considering a lawsuit against Microsoft/OpenAI. 

tl;dr:

*   Copilot is trained on open source software.
*   Copilot doesn't respect the licensing agreements of that software.
*   Copilot doesn't have a clear fair use argument for doing so.
*   By accepting copilot suggestions, you are potentially violating licensing agreements yourself.

Sections of particular interest:

> \[W\]e inquired pri­vately with Fried­man and other Microsoft and GitHub rep­re­sen­ta­tives in June 2021, ask­ing for solid legal ref­er­ences for GitHub’s pub­lic legal posi­tions … They pro­vided none.

\- Software Freedom Conservancy

> *“You are respon­si­ble for ensur­ing the secu­rity and qual­ity of your code. We rec­om­mend you take the same pre­cau­tions when using code gen­er­ated by GitHub Copi­lot that you would when using any code you didn’t write your­self. These pre­cau­tions include rig­or­ous test­ing, **IP \[(= intel­lec­tual prop­erty)\] scan­ning** \[my emphasis\], and track­ing for secu­rity vul­ner­a­bil­i­ties.”*

\- https://docs.github.com/en/copilot/overview-of-github-copilot/about-github-copilot#using-github-copilot
# Untapped Potential at 13-18

The Western world is facing a crisis of **stretched adolescence,** keeping children exclusively "learning" for longer than necessary. 

Imagine that you are a student, around 14 years old. You fall into the top 5%, not the extremely motivated startup founder or math prodigy that you hear about on the news, but comfortably more intelligent and curious than the others. The system sucks some of your free time with busywork, but you build up a knowledge base, and start working on small projects related to your area of interest. Now what?

Young people are capable. Their neurons are fully myelinated, their worldview is fresh, their passion is strong. But we don't use them.

In fact we make it difficult for them to thrive: labor laws make it bothersome for companies to take on anyone under 18, even if they can demonstrate competence, so that path is rocky. Research internships and freelancing are more viable, but still extremely underused. Perhaps it's because the school system drowns people in busywork, forcing them to study things just to forget them within a few days,[^dx0teym9xi9] and personal projects are stifled under the workload. Competent teens fall into apathy because the work is too easy, application of anything more difficult is inaccessible, and there's only so much knowledge-gathering they can do before they want to **do something**.

We need to put frameworks in place to take advantage of capable high school students. The current assumption is that the most qualified will find a way, and while that may be true, that leaves behind a huge untapped resource. Not everyone is hyper-ambitious but if provided engaging work or more exposure to interesting problems, they can be both useful and satisfied, and the world benefits.

The Internet provides some of these opportunities, however working with "professionals" on larger problems is an experience that cannot be replicated. The combination of years of wisdom and accumulated knowledge with a modern view of a problem without the inset biases of current models has serious potential.

TL;DR: Teenagers are underappreciated and a vast and unique source of cognitive power that should be more cleanly introduced into the working world.

[^dx0teym9xi9]: https://en.wikipedia.org/wiki/Forgetting_curve
# Should we push for requiring AI training data to be licensed?

We frequently speak about AI capability gain being bad because it shortens the timeframe for AI safety research. In that logic, taking steps to decrease AI capability would be worthwhile.

At the moment the large language models are trained with a lot of data without the company, that trains the language model, licensing the data. If there would be a requirement to license the required data, that would severely reduce the available data for language models and reduce their capabilities. 

It's expensive to fight lawsuits in the United States. Currently, there are artists who feel like their rights are violated by Dalle 2 using their art as training data. Similar to how Thiel funded the Gawker lawsuits, it would be possible to support artists in a suit against OpenAI to require OpenAI to license images for training Dalle 2. If such a lawsuit is well-funded it will be much more likely that a precedent for requiring data licensing gets set which would slow down AI development. 

I'm curious about what people who think more about AI safety than myself think about such a move. Would it be helpful?
# Scaling Laws for Reward Model Overoptimization

**TL;DR**: Reward model (RM) overoptimization in a synthetic-reward setting can be modelled surprisingly well by simple functional forms. The coefficients also scale smoothly with scale. We draw some initial correspondences between the terms of the functional forms and the Goodhart Taxonomy. We suspect there may be deeper theoretical reasons behind these functional forms, and hope that our work leads to a better understanding of overoptimization.

Some other results:

*   We compare two different methods of optimization (RL, best-of-n); RL consumes much more KL distance than best-of-n for the same amount of optimization. 
*   We show that using KL distance between the initial and optimized policies is not a reliable measure of optimization power when comparing different methods. We also find that penalizing based on the KL distance in RL does not change the KL distance--gold reward frontier in our setting.
*   We find some very preliminary evidence that at our scales, scaling the policy does not substantially increase the amount of optimization pressure placed on the RM. Further study of this effect could be relevant to some models of inner optimization.
*   With a few additional assumptions, our functional form also makes some predictions about iterated RLHF (that it will reduce Extremal Goodhart but not Regressional Goodhart).
*   This setup only captures the effect of overoptimizing a learned RM relative to using the ground truth directly. In particular, this setup does not directly capture any mismatch between the ground truth labels and the human intent, which plausibly contains a majority of the difficulty of outer alignment

*If you're interested in the intersection of alignment theory and empirical research, we're hiring! We want to gain insight on things like Goodhart's Law, ELK, and inner alignment via experiments on large language models. Shoot me (leogao) a DM if you're interested.*

### Select figures from the paper

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7b4129fd8a16887b7147db7031a3ff7e8f282b2902713ddd.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c8148998e539e969db230203c4c587618bdd84007dbc49d4.png)
# What Does AI Alignment Success Look Like?

Suppose you are put in stasis and wake up 10 years after the FOOM. You are trying to figure out if the AI Alignment project succeeded or not. How can you tell? Not vaguely, concretely. What metrics indicate success, and what metrics indicate failure? The following are potential examples, based on various sci-fi AI tropes. No need to discuss each one separately, there are millions more, the idea is to delineate failures from success in the general case.

*   Earth has been turned into a mega-brain with no visible humans around.
*   The Universe around the Solar System has disappeared (or turned into an apparently solid shell of unknown composition), but humans are still around, living in abundance of anything they want or need and seemingly content to stay that way.
*   Every human gets their own (real or simulated) universe to play with. Some end up creating trillions of creatures whom they torture for fun.
*   Humanity lives inside a giant simulation.
*   AI development is under a strictly enforced interdict.
*   Anything remotely alive-looking is digimon-like creatures, who proclaim themselves human.
*   The universe appears empty except for one black hole that encodes humanity (in some form) in its horizon microstates.

Any links discussing this would be appreciated, too.
# Rhythmic Baby Toys

In another [debatably educational](https://www.jefftk.com/p/infant-talk-boxes) application of my [rhythm stage setup](https://www.jefftk.com/news/jammer), Nora really likes playing with my bass pedals. She especially likes them when I have them set in a particular way:

*   When you tap them they make a noise
    
*   When you tap them with a consistent beat they make them more complex noise.
    
*   If your beat isn't consistent enough they go back to the simpler noise until your tempo is clear again.
    

Here's a demo:

This has me wondering whether some sort of toy like this could be good at teaching kids a sense of consistent rhythm? The initial noise is enough to make the toy fun, but then if you can be consistent enough you get something more fun. You could probably start with Somewhat relaxed threshold for "consistent enough", and then tighten it over time as they got better?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02p8zx2NKeYidwp7pg2uE98d52fKTewUPNZzdB72kHuLG1io9bAzYuGZN7amEP79jbl)*
# AI Research Program Prediction Markets

John Wentworth [requested that someone would create prediction markets for various AI research programs](https://www.lesswrong.com/posts/oxSX9XDQHLu5YLpaD/how-to-make-prediction-markets-useful-for-alignment-work?commentId=cbFt4p9Rwwwvnof2u). I've now done so, for four programs:

Remember, though - [Measure Lots Of Stuff](https://www.lesswrong.com/posts/9kNxhKWvixtKW5anS/you-are-not-measuring-what-you-think-you-are-measuring#Solution__Measure_Lots_of_Things): It would probably be good if other people made prediction markets that measured things from other angles. Make sure that you read [John Wentworth's original post](https://www.lesswrong.com/posts/oxSX9XDQHLu5YLpaD/how-to-make-prediction-markets-useful-for-alignment-work) to know what is needed by alignment researchers.
# Rough Sketch for Product to Enhance Citizen Participation in Politics

For politics and governance in the US, does there exist a tool that:

*   Prompts the user to enter their rough location (e.g., town, county, state)
*   Prompts the user to select interest keywords (e.g., housing, animal welfare, cannabis) 
*   Lists *pending* local (town / county), state, and federal level laws and regulations w.r.t. these interests
*   Lists *current* local (town / county), state, and federal level laws and regulations w.r.t. these interests
*   Includes summaries of *current* local (town / county), state, and federal level laws and regulations for accessibility
*   Lists special interest groups that typically support/counter laws and regulations relating to these interests at the local (town / county), state, and federal level
*   Lists ways in which the user could influence laws and regulations w.r.t. these interests (e.g., a step-by-step tutorial for participating in a certain election) 

If this tool doesn't exist, how much value would people get from it if it existed? How difficult would it be to implement each part? (please point me to any tools / organizations that roughly fulfill the duties outlined in the bullet points)

Lastly, w.r.t. the point 

*   Includes summaries of *current* local (town / county), state, and federal level laws and regulations for accessibility

I think GPT-X might work well for summarizing and distilling legal language - has this been done already?
# When apparently positive evidence can be negative evidence

This is a thought-provoking journal article discussing cases where:

1.  You have something with two possible states, A and B, and you can perform fairly accurate, mostly independent tests which distinguish between A and B.
2.  However, there is some failure case with a low prior probability, in which the actual state may be B, but something confounds all your tests and makes them look like they favor A regardless.
3.  If you then perform many tests, and they all favor A, the first few tests will make you update towards A, but then if you keep performing more tests whose results favor A, you will actually start updating towards B, because you will start to think you are in the failure case.

An interesting anecdote which I haven't verified:

> Under ancient Jewish law one could not be unanimously convicted of a capital crime—it was held that the absence of even one dissenting opinion among the judges indicated that there must remain some form of undiscovered exculpatory evidence.

Via [Hacker News](https://news.ycombinator.com/item?id=33278349).
# Crossword puzzle: LessWrong Halloween 2022

Just in time for Halloween - the spookiest themed crossword puzzle of the year!

* [Interactive web version](https://gamesforthebored.github.io/crossword/crossword-lw-halloween-2022.html)
* [Printable PDF](https://gamesforthebored.github.io/crossword/crossword-lw-halloween-2022.pdf)

The comments below may contain unmarked spoilers. I don't expect many people will solve it without doing at least some Wikipedia lookups, but you're welcome to try.

Enjoy!
# Deepfake(?) Phishing

\[EDIT: I now think this probably wasn't a deepfake, but a recording the attacker made when using the same attack against the previous person. See the comments.\]

I think someone just tried to phish my Facebook account, including a fake video of a FB friend. Here's the conversation: 

Them, via FB Messenger, 9:32am:

Please ,I was trying to login in my instagram page on Facebook my new phone and they ask me to find someone to help me receive a code, Facebook gave me two friends suggestions and you one of them, the other person isn't online. will you Help me receive the code please?

Me:

I'm sorry you're having trouble logging in! Just so I can make sure your account hasn't been hacked, how did we meet?

Them:

\[Calls me over FB Messenger, audio isn't working but it does look like them. I'm completely convinced at this point.\]

Me:

Audio wasn't working, but I did recognize you  
What do you need me to do?

32665, over SMS:

NNNNNNNN is your Facebook password reset code \[this number has previously sent me FB resets\]

Them:

Send me the code sent to you minute ago

Me:

Hmm, those look like the code to reset the password to my account?  
Can we call again?

Me:

\[I try to call them back, doesn't go through\]

Them:

Nahh it's for my instagram

Them:

Having bad connections here

Them:

Send me the code ?

Me:

sorry, I'm still worried your account has been hacked -- can we do another call?

Them:

\[Calls me over FB Messenger, audio is still not working, and the video feels slightly off. Ends quickly on their end. Possibly it's even the same video from last time?\]

Me:

We're you able to hear me?

Them:

My connections

I've reported their account as hacked.

Things that made me suspicious:

I don't think FB has any sort of account recovery that looks like this

This is exactly what an attempt to hack my FB account would look like

9:30am, even though that makes it 6:30 where they live

Video call didn't have any audio

They couldn't receive incoming video calls

Text did't feel like them, though I don't know them that well.

Here's a screenshot I took during the second video call:

[![](https://www.jefftk.com/deepfake-phishing-video-call.jpg)](https://www.jefftk.com/deepfake-phishing-video-call-big.jpg)

Even with all those things, the video call would normally have been very convincing, and it did briefly convince me. I could easily see it fooling someone who didn't know about deepfake video.

*Comment via:* [*facebook*](https://www.facebook.com/jefftk/posts/pfbid02kXY1EtP1mqUxs6fKuoVRwtYzN7dPMWYCDDs1v6igxsFXENaV1xjS9uWDkzqm7Hfwl)
# Moorean Statements

*Moorean statements* are statements like:

> It's not raining but I believe it is.

These statements sound strange because any agent that outright *tells you* it's not raining must already at least *tacitly* represent that fact in their world model. That agent is plugged into their world model well enough to report what it says, but not well enough to accurately model their model. For this explicit a Moorean statement, the epistemic strangeness is so obvious that basically no one will have that combination of access to and confusion about their world model.

An Eliezerism my Eliezer-model often generates is that *many social scripts involve expressing Moorean propositions.* They're subtler, but the essential confusion is the same.

> I'm a committed Christian because my parents are -- that's just how I was raised.
> 
> Well, if intuitions aren't epistemically admissible in philosophy, philosophers would be out of a job!

What? How can you *simultaneously* recognize the non-epistemic generator of your belief *and* hold the belief?

Can you generate more instances?
# The highest-probability outcome can be out of distribution

"Out of distribution" is a phenomenon where a data point x does not come from the distribution you had in mind. To give some examples, here is what might be considered OOD:

*   You play a dice game against a cheater, and the cheater uses a die that has been weighted to always roll six. Thus their rolls look like 666666..., whereas your rolls look like 312265....
*   You order an apple from a store and the store decides to drive a truck into your house in order to murder you.
*   You are reading an interesting post on LessWrong, and th2u  
    mCXh4:xaapdQA  
    r bKv4ztPktiR)pYpXwT-j  
    %?:Ta4Or  
    XXdicVAQ?;DHJFh?U4wLW! iCmVD~z1iQT5cB4x&J_A-2MmUxCCxDdA3hYXjomA "m5mID;9-C9~=%~-j6ba"2N!fV~jikXZOQ2iEsnNxR0nggVYR:ZF:9&p  
    m#R"Xf  
    j~=vFiM_N(&~M%8amq2q5z5

Possibilities 2 and 3 seem to be based on something that is unlikely or surprising happening, so they seem to suggest that being "out of distribution" is about being low-probability. But if you think about it for possibility 1, then every sequence of dice rolls of equal length should have the same probability when using fair dice, and therefore low probability cannot account for possibility 1.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4dd48cdb75ae780b9c01869871f8f879044608f5bf0e620b.png)

In fact, we can construct a scenario where a point is "out of distribution" despite being the highest-probability point in the distribution. Imagine a gaming place where everyone is expected to cheat, playing with dice that have 58% chance of rolling a 6. In that case, an ordinary roll sequence would look like 624646662443.... However, if someone got a roll sequence full of 6s, like 666666666666..., then their rolls would be out of distribution, and it would look like they were cheating beyond the accepted weighted dice.

Conceptually, what is happening here is that even though 666666666666... is the most likely roll sequence, there are so many other sequences of rolls that have fewer 6s that their lower likelihood is counterbalanced (and then some) by their greater number. The sum of the probabilities of other sequences ends up being greater than the probability of 666666666666....

However, this doesn't explain the full story. After all, the sum of the probabilities of the non-624646662443... roll sequences is also much greater than the probability of 624646662443.... *Any* one specific roll sequence is unlikely, so why do we consider some to be "out of distribution" and others not?

This is probably not a fully solved problem. Perhaps in a later post, I will give one piece that I believe to be relevant. But until then, here's a relevant meme:

![Image](https://pbs.twimg.com/media/FFd4ahHUYAInM_y?format=jpg&name=medium)

*Thanks to Justis Mills for proofreading and feedback.*
# Have you noticed any ways that rationalists differ? [Brainstorming session]

I have a severe psychometrics addiction, so I've regularly been tempted to create some sort of "rationalist type test" or similar, which rationalists could take to classify themselves compared to other rationalists. I particularly like to create such tests using data-driven methods, such as factor analysis. But factor analysis requires a bunch of primitive variables to summarize. And I thought, the most logical way to get such variables is to ask people what they've already come up with.

So, do you have any thoughts on ways that rationalists differ from each other, or ways that rationalists differ from nonrationalists, or perhaps even ways that nonrationalists differ from each other in ways that might be interesting to put into a test for rationalists?

Also, if you don't have a LessWrong account, or want to share this question with rationalists who don't hang out on LessWrong, [I've created a Google Form that you can fill out instead](https://docs.google.com/forms/d/e/1FAIpQLSfs0j_KWcDA5WanFVS-lsYQT_pH0W7__BMkuLZpooYk96ZWIQ/viewform). If you've got something spicier that you would prefer doesn't become public, then [I have also created a private version of the form here](https://docs.google.com/forms/d/e/1FAIpQLSeTOYCFuZJX8-QpZT1_Hd_Ms_ibyrhJR17jubIORCMc1sy5EQ/viewform).

I don't know how useful my analysis could become, but at least in theory it seems like it could become more generally useful. For instance, [it might be useful for providing targeted advice to certain subgroups of rationalists](https://slatestarcodex.com/2013/06/09/all-debates-are-bravery-debates/).

Edited to add: A lot of the comments seem to be bringing up abstract, general traits that have been studied in differential psychology. Certainly those could also be interesting to investigate rationalist on, but I have numerous philosophical objections to many of the traits, and I would instead like to design a new test with traits specific to the rationalist community, in terms of more concrete things that rationalists do and experience.