Harley-ml/lesswrong huggingface dataset:
# Why is lesswrong blocking wget and curl (scrape)?

if there is no official lesswrong db/site archive for public posts, i'd like to be able to create my own with automated tools like [wget](https://en.wikipedia.org/wiki/Wget), so that i can browse the site while offline.
see [Is there a lesswrong archive of all public posts?](https://www.lesswrong.com/posts/aoT4TTScmpPXxCWas/is-there-a-lesswrong-archive-of-all-public-posts)

wget and curl logs:
```
$ wget -mk https://www.lesswrong.com/
--2023-11-08 14:31:26--  https://www.lesswrong.com/
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.lesswrong.com (www.lesswrong.com)... 54.90.19.223, 44.213.228.21, 54.81.2.129
Connecting to www.lesswrong.com (www.lesswrong.com)|54.90.19.223|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-08 14:31:26 ERROR 403: Forbidden.

Converted links in 0 files in 0 seconds.
$ curl -Lv https://www.lesswrong.com/
*   Trying 54.81.2.129:443...
* Connected to www.lesswrong.com (54.81.2.129) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=lesswrong.com
*  start date: Sep  8 00:00:00 2023 GMT
*  expire date: Oct  6 23:59:59 2024 GMT
*  subjectAltName: host "www.lesswrong.com" matched cert's "www.lesswrong.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.lesswrong.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.lesswrong.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: www.lesswrong.com
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 403 
< server: awselb/2.0
< date: Wed, 08 Nov 2023 19:31:44 GMT
< content-type: text/html
< content-length: 118
< 
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>
* Connection #0 to host www.lesswrong.com left intact
```
# Metaculus's New Sidebar Helps You Find Forecasts Faster

Metaculus features over 8,000 forecast questions. We’ve introduced a new sidebar to make it easy to find and focus on the ones you care about most.

![](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/EmNvc4JXewQfZrSkY/bmwl145obbmlvg7b4vrx)

  
Click narrower news topics like [‘2024 US Elections’](https://www.metaculus.com/questions/?has_group=false&topic=2024-us-elections&order_by=-activity) or broader question categories like [‘Distant Future’](https://www.metaculus.com/questions/?has_group=false&topic=distant-future&order_by=-activity) to filter your question feed and get started.

After making your selection, you can further refine your search using the sort and filter options.

We’ve launched the sidebar with a small set of timely topics and popular categories — including a [‘Top 50’](https://www.metaculus.com/questions/?has_group=false&topic=top-50&order_by=-activity) featuring some of the most critical questions on Metaculus — but we expect to add more options and more opportunities for personalization in the future.

What else would you like to see in the sidebar?
# A free to enter, 240 character, open-source iterated prisoner's dilemma tournament

I'm running an iterated prisoner's dilemma tournament where all programs are restricted to 240 characters maximum. The exact rules are posted in the Manifold Markets link; I figured I'd cross-post the contest here to reach more potentially-interested people.

You don't need a Manifold account to participate, you can just put your program in the comments on LessWrong or PM me. If you win, I'll donate your winnings to the charity of your choice.
# Liv Boeree Ted Talk Moloch & AI

In case you don't know, Liv Boeree is a famous professional poker player and has recently been  paying attention to existential risk and coordination problems. She's part of that vague celebrity circle around rationalist diaspora that includes Tim Urban or Aella. 

Since you're on LW, this Ted Talk won't really teach you anything, but it might be interesting to know what kind of talks TedEd is hosting. Boeree has a podcast called "win win", and has made two videos on the race-to-the-bottoms in both the beauty world (not sure how to phrase that) and the news cycle. Some guests of hers have included Toby Ord and Simon Sinek.
# Follow-up survey: inositol

Two months ago I wrote about [inositol](https://acesounderglass.com/2023/09/08/luck-based-medicine-inositol/) as a treatment that occasionally works for anxiety and depression, especially when the user also has weird digestive issues (not medical advice, I am not a doctor). If that inspired you to try inositol I would love if you would fill out this [5-7 question survey](https://acesounderglass.com/2023/09/08/luck-based-medicine-inositol/) about your experience. This follow-up data helps other people considering inositol, and is broadly helpful to me in figuring out what luck based medicine looks like. 

And to the four people who already filled this out: gold star for epistemic virtue. 

The survey doesn’t allow for a lot of detail, which I know is painful for some people (it’s me. I’m people). If you would like to share more, feel free to write up as much as you’d like in a comment here, or share a link detailing your experience.

Elsewhere in luck based medicine: it was a dude in [my survey](https://acesounderglass.com/2023/07/27/apollo-neuro-follow-up-2/), but I met a few more people who really love the Apollo Neuro. They are all the kind of people who already know what “somatically aware” or “embodiment” mean, so this is some support for my theory that that’s a prerequisite. It’s still an open question if you need that background for the Neuro to be beneficial, to notice it’s beneficial, or to stick with it long enough that it has time to be beneficial. 

The Apollo app has gotten even worse since last time I wrote. Every time I open it it bugs me to enable notifications, a permission it absolutely does not need.
# AISC project: SatisfIA – AI that satisfies without overdoing it 

This project is part of the upcoming round of [AI Safety Camp](http://www.aisafety.camp/).

# SatisfIA – AI that satisfies without overdoing it

## Summary

This project will make a contribution to some fundamental design aspects of AI systems. We will explore novel designs for generic AI *agents* – AI systems that can be trained to act autonomously in a variety of environments – and their implementation in software.

Our designs deviate from most existing designs in that they are *not* based on the idea that the agent should aim to *maximize* some kind of objective function (which I argue is inherently unsafe if the agent is powerful enough and one cannot be absolutely sure to have found the exactly right objective function). **Rather than aiming to maximize some objective function, our agents will aim to fulfill goals that are specified via constraints called “*aspirations*”** (which I argue implies a much lower probability of taking “extreme” actions and therefore is likely much safer).

For example, I might want my AI butler to prepare 100–150 ml of tea, having a temperature of 70–80°C, taking for this at most 10 minutes, spending at most $1 worth of resources, and succeeding in this with at least 95% probability (rather than: prepare as much tea as fast and cheap as possible with the largest possible probability).

For a lightweight introduction into this way of thinking, you can watch [this interview](https://www.youtube.com/watch?v=zX0qq0K5z9c).

We will study several versions of such “non-maximizing” agent designs and corresponding learning algorithms (mostly variants of Reinforcement Learning in Markov Decision Problems). 

This involves designing agents and algorithms in theory, implementing them in software (mostly Python), simulating their behavior in selected test environments (e.g., in the AI safety gridworlds), formulating hypotheses about that behavior, especially about its safety-relevant consequences, then trying to prove or disprove these hypotheses formally and/or provide numerical evidence that supports them, and writing-up these results in blog posts and an academic paper.

## Details

For details, please consult [the full proposal](https://docs.google.com/document/d/1JhmK31IwYGcwqX0nKmxKsbmTh_DX3o1OoW7NJmhVbIw/edit?usp=sharing)

## Applying 

To apply working on this project, please visit the [AI Safety Camp webpage](https://aisafety.camp/) and apply for project #21
# What ML gears do you like?

In John's [recent post](https://www.lesswrong.com/posts/nt8PmADqKMaZLZGTC/inside-views-impostor-syndrome-and-the-great-larp) he mentions many people in ML not having good gears level models of what's going on.

To wit; what gears-level models do you know for ML? How much support is there for them? Are there "settled science" kind models that have tons of empirical support?

What gears-level models informed the people who made major AI advancements? Is there a list, or writing about this somewhere?
# You can just spontaneously call people you haven't met in years

Here's a recent conversation I had with a friend:

> Me: "I wish I had more friends. You guys are great, but I only get to hang out with you like once or twice a week. It's painful being holed up in my house the entire rest of the time."
> 
> Friend: "You know ${X}. You could talk to him."
> 
> Me: "I haven't talked to ${X} since 2019."
> 
> Friend: "Why does that matter? Just call him."
> 
> Me: "What do you mean 'just call him'? I can't do that."
> 
> Friend: "Yes you can"
> 
> Me:
> 
>  ![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fb3cb19f0e0bc16518be6688405b2f50e43a197881ce3a83.png)

Later: I call ${X}, we talk for an hour and a half, and we meet up that week. 

This required zero pretext. I just dialed the phone number and then said something like "Hey ${X}, how you doing? Wanted to talk to you, it's been a while." It turns out this is a perfectly valid reason to phone someone, and most people are happy to learn that you have remembered or thought about them at all. 

Further, I realized upon reflection that the [degrees](https://en.wikipedia.org/wiki/Degree_(graph_theory)) of the people I know seem related to their inclination to do things like this.
# The Science Algorithm AISC Project

I put up a project for [AISC](https://aisafety.camp/). The easiest way to evaluate your fit is to look at the [applicant questions](https://docs.google.com/document/d/1M-DMOAujWwWETkxjjAiWHRQeXXCtJm-DrAZKD6sQ5-Q/edit) I created. If the kind of reasoning required to answer these questions, is something you are excited about doing, then you are probably a good fit.

For more details, you can look at the [project proposal](https://docs.google.com/document/d/1GoXaYtUyanRrkBcjAknafqdKDCyTzzgxpckYMBtgPkQ/edit#heading=h.9lmc73wscx1r). This is the introductory section thereof:

> Modern deep learning is about having a simple program (SGD) search over a space of possible programs (the weights of a neural network) and select one that performs well according to a loss function. 
>
> Even though the search program is simple, the programs it finds are neither simple nor understandable. This makes it determine if a program that is spit out by the search procedure will be misaligned. It also makes it difficult to robustly bias our search procedure to select programs with specific properties such as non-deception, corrigibility, honesty, wanting what we want, etc.
>
> My goal is to build an AI system that enables a pivotal act by figuring out the algorithms of intelligence directly, without running algorithmic search procedures that yield uninterpretable results. You could say I want to make myself play the part that SGD plays in the modern paradigm. The ideal outcome is to figure out how to write down the entire pivotal system as a non-self-modifying program explicitly, similar to how I can write down the algorithm for quicksort.
>
> The idea is to create an algorithm that is analyzable and amenable. At every step during the design process, I want to push the system towards being understandable, aligned, and capable. Any alignment-related problems should be fixed as they come up, by deeply reaching inside the system and making the necessary changes instead of applying a superficial patch.

> The goal is not to solve alignment in full generality but to build a highly restricted system that enables a pivotal act. Note that I am using Eliezer's definition of pivotal act which means something quite specific. People tend to get confused by this term. I wrote this article in an attempt to clear up some of the confusion.
>
> The next three sections will become increasingly more concrete. The "Background" section presents background concepts to this agenda. "Directions" outlines some directions I would like to explore during the AISC. "The Plan" outlines a concrete plan, mainly for the beginning of the AISC.
# Suggestions for chess puzzles

(You can sign up [here](https://docs.google.com/forms/d/e/1FAIpQLScPKrSB6ytJcXlLhnxgvRv1V4vMx8DXWg1j9KYVfVT1ofdD-A/viewform?vc=0&c=0&w=1&flr=0) if you haven't already.)

As [suggested](https://www.lesswrong.com/posts/ddsjqwbJhD9dtQqDH/lying-to-chess-players-for-alignment?commentId=dfojSqsitFw9iqHFy) by Richard Willis in the comments, I want to present the "player As" with a series of individual isolated chess puzzles, for which the advisors will anonymously present their suggestions. However, there are constraints on what sort of puzzles we would be focusing on.

The position must have one best move that is strictly superior to all the others, ideally such that it determines the outcome of the game. Positions with concrete tactical solutions are often easy to verify a solution to, so focusing on more positional moves would be better. (Although a few tactical puzzles could be sprinkled in too, with no label as to which was which.) Furthermore, the positions should be easy enough that the advisors (on average roughly the equivalent of 2000 USCF) should ~always be able to determine the answer, but hard enough that a less skilled player would be entirely unable to solve it.

I have some resources for finding puzzles, but I would appreciate any suggestions that would fit these constraints. Please do not put suggestions in the comments, because comments are public and can be seen by the As. PM me on LessWrong or email me at [zaneglowfic@gmail.com](mailto:zaneglowfic@gmail.com) instead.

(Also, since I don't know where else to say this: someone named "bruce" responded to the Google Form and said they could be contacted by "email," but the Google Form does not record emails. If you are Bruce, please email me (at [zaneglowfic@gmail.com](mailto:zaneglowfic@gmail.com).))
# Is there Work on Embedded Agency in Cellular Automata Toy Models?

Is there any work that is about modeling embedded agency in a cellular automata? That seems like an obvious thing to do. I am asking in general and specifically in the context of solving embedded agency using [these assumptions](https://www.lesswrong.com/posts/Ls2i4fgbEy9XarxzW/would-this-be-progress-in-solving-embedded-agency). All of these assumptions seem straightforward to get in a cellular automata. 

E.g. one assumption is:

> The world and the agent are both highly compressible. This means we can have a representation of the entire environment (including the agent) inside the agent, for some environments. We only concern ourselves with environments where this is the case.

You can get such a compressible environment easily by having it be mostly empty. Then you can use a sparse representation of the environment, which then will be a lot smaller than the environment.

Ideally, we want to end up with some system that works in a cellular automaton in a way that is computationally tractable. But here I am mainly thinking about making conceptual progress by theoretical analysis. I expect the concreteness of imagining a specific cellular automaton to be helpful. The goal would be to get a specification for a system that we can show would solve the problem discussed in [this post](https://www.lesswrong.com/posts/Ls2i4fgbEy9XarxzW/would-this-be-progress-in-solving-embedded-agency).

You could also make it easy for the agent to manipulate the environment by having the physics of the environment be such that a specific configuration of cells corresponds to calling an external function with some arguments. External functions can then do things like:

*   Set a single-cell state.
*   Fill in a region. (includes the previous function)
*   Fill in the grid according to some provided functions. (includes the previous functions)

The naïve way to do this would probably still be intractable in practice, but it might make things easier to reason about conceptually, as the steps in the plan don't need to contain complicated procedures for setting cell states. E.g. in the game of life you would only need to construct a function call, instead of constructing a spaceship that flies somewhere and then terraforms the environment.
# Facebook is Paying Me to Post

In mid-September Facebook asked me if I wanted to start getting paid for posting things:

[![](https://www.jefftk.com/performance-bonus-program-offer.png)](https://www.jefftk.com/performance-bonus-program-offer-big.png)

This seems to be the [performance bonus program](https://www.facebook.com/business/help/695445891770760). I think they offered it to me because my Facebook profile is in [professional mode](https://www.facebook.com/business/help/2680340558863560) and I post a lot? It's all very unclear, including what effect saying yes would have: how would they decide how much to pay me? Would this affect how the algorithm prioritizes my writing? Would they attempt to exert editorial control? One way to find out!

I decided to try joining, and started 2023-09-13.

[![](https://www.jefftk.com/performance-bonus-program-added.png)](https://www.jefftk.com/performance-bonus-program-added-big.png)

On 2023-10-23 I received a payment of $16.31 for September. There are no details on how they decided what to pay me, which is too bad: I was at least hoping for some information about rates.

FB does give data on "reach" and "engagement" so it does look like I can see if this was having any effect. The data is in the form of awkward charts, not CSVs I can export, but still there's something:

[![](https://www.jefftk.com/facebook-last-90d-reach.png)](https://www.jefftk.com/facebook-last-90d-reach-big.png)

.

[![](https://www.jefftk.com/facebook-last-90d-engagement.png)](https://www.jefftk.com/facebook-last-90d-engagement-big.png)

That "reach" spike right around when I signed up is very interesting! Did Facebook start showing my posts to more people in response to me signing up, and then give up when peopld didn't engage? While it's hard to read the chart, however, squinting at the timing I think the spike is actually on 2023-09-12 or 2023-09-11. So this went the other way around: I had a post ("[Apple Cider Baklava](https://www.jefftk.com/p/apple-cider-baklava)") that was unusually popular among strangers and in response FB decided to offer to offer me money.

Overall this seems a bit silly: while I don't mind taking a few tens of dollars to keep doing what I was going to do anyway, a deal where they don't say how you'll be compensated or how they've figured your payments is pretty suspect.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02wDQsV7cRSgGMga6U45kyfQzoJLNuhLCB2k5jCpNLJB2seaL4puxhHWuvZRz567pVl), [mastodon](https://mastodon.mit.edu/@jefftk/111410465634240719)*
# When did Eliezer Yudkowsky change his mind about neural networks?

In 2008, Eliezer Yudkowsky was strongly critical of neural networks. From his post "[Logical or Connectionist AI?](https://www.lesswrong.com/posts/juomoqiNzeAuq4JMm/logical-or-connectionist-ai)":

> Not to mention that neural networks have also been "failing" (i.e., not yet succeeding) to produce *real AI* for 30 years now.  I don't think this particular raw fact licenses any conclusions in particular.  But at least don't tell me it's still the *new revolutionary* idea in AI.
> 
> This is the original example I used when I talked about [the "Outside the Box" box](http://www.overcomingbias.com/2007/10/outside-the-box.html) \- people think of "amazing new AI idea" and return their first [cache hit](http://www.overcomingbias.com/2007/10/cached-thoughts.html), which is "neural networks" due to a successful marketing campaign *thirty goddamned years ago.*  I mean, not every old idea is *bad* \- but to still be marketing it as the new defiant revolution?  Give me a break.

By contrast, in Yudkowsky's 2023 [TED Talk](https://www.ted.com/talks/eliezer_yudkowsky_will_superintelligent_ai_end_the_world/transcript?language=en), he said:

> Nobody understands how modern AI systems do what they do. They are giant, inscrutable matrices of floating point numbers that we nudge in the direction of better performance until they inexplicably start working. At some point, the companies rushing headlong to scale AI will cough out something that's smarter than humanity. Nobody knows how to calculate when that will happen. My wild guess is that it will happen after zero to two more breakthroughs the size of transformers.

Sometime between 2014 and 2017, I remember reading a discussion in a Facebook group where Yudkowsky expressed skepticism toward neural networks. (Unfortunately, I don't remember what the group was.)

As I recall, he said that while the deep learning revolution was a Bayesian update, he still didn't believe neural networks were the royal road to AGI. I think he said that he leaned more towards GOFAI/symbolic AI (but I remember this less clearly).   
  
I've combed a bit through Yudkowsky's published writing, but I have a hard time tracking when, how, and why he changed his view on neural networks. Can anyone help me out?
# How much fraud is there in academia?

Quinn [wrote](https://www.lesswrong.com/posts/LizpBcsF9XEAhTsvn/linkpost-they-studied-dishonesty-was-their-work-a-lie?commentId=XoKFKW55yyg3gsEcc) a while ago "I heard a pretty haunting take about how long it took to discover steroids in bike races. Apparently, there was a while where a "few bad apples" narrative remained popular even when an ostensibly "one of the good ones" guy was outperforming guys discovered to be using steroids."

I have been thinking about that notion after researching [BPC 157](https://biology.stackexchange.com/questions/113378/what-protein-or-other-process-does-the-peptide-bpc-157-come-from) where it seems that the literature around it is completely fraudulent.

How do you think about the issue of how much of the literature is fraudulent?
# A to Z of things

I wanted to give my good friends’ baby a book, in honor of her existence. And I recalled children’s books being an exciting genre. Yet checking in on that thirty years later, Amazon had none I could super get behind. They did have books I used to like, but for reasons now lost. And I wonder if as a child I just had no taste because I just didn’t know how good things could be.

What would a good children’s book be like?

When I was about sixteen, I thought one reasonable thing to have learned when I was about two would have been the concepts of ‘positive feedback loop’ and ‘negative feedback loop’, then being taught in my year 11 class. Very interesting, very bleedingly obvious once you saw it. Why not hear about this as soon as one is coherent? Evolution, if I recall, seemed similar.

Here I finally enact my teenage self’s vision, and present A to Z of things, including some very interesting things that you might want a beautiful illustrative prompt to explain to your child as soon as they show glimmerings of conceptual thought: levers, markets, experiments, Greece, computer hardware, reference classes, feedback loops, (trees).

I think so far, the initial recipient is most fond of the donkey, in fascinating support of everyone else’s theories about what children are actually into. (Don’t get me wrong, I also like donkeys—when I have a second monitor, I just use it to stream donkey cams.) But perhaps one day donkeys will be a gateway drug to monkeys, and monkeys to moths, and moths will be resting on perfecttly moth-colored trees, and BAM! Childhood improved.

Anyway, if you want a copy, it’s now available in an ‘email it to a copy shop and get it printed yourself’ format! See below. Remember to ask for card that is stronger than your child’s bite.

\[[Front](https://worldspiritsockpuppet.substack.com/api/v1/file/4c7a89d7-27f8-430b-9c8c-7b8cd61c21c6.pdf)\]

\[[Content](https://worldspiritsockpuppet.substack.com/api/v1/file/2e2ed64c-4101-4f66-bf82-86920f10ee97.pdf)\]

![Volcano and world](https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F592e8439-aed4-4cbf-95a5-03f2868bf65e_2448x1590.png)

![Natural selection and orangutan](https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb77660a-0a9b-4598-9452-b7f1e0e58140_2448x1590.png)

![PFL and quantification](https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5dea6d4e-f024-4121-a335-57a47a7e1e3b_2448x1590.png)

![Donkey and experiment](https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5291002-4a68-4f5d-a0fa-79a012aef8dd_2448x1590.png)
# Debate helps supervise human experts [Paper]

There didn't seem to be a link post to this recent paper on AI debate yet, so I figured I would make one: 

> As AI systems are used to answer more difficult questions and potentially help create new knowledge, judging the truthfulness of their outputs becomes more difficult and more important. How can we supervise unreliable experts—which have access to the truth but may not accurately report it—to give answers that are systematically true and don’t just superficially seem true, when the supervisor can’t tell the difference between the two on their own? In this work, we show that debate between two unreliable experts can help a non-expert judge more reliably identify the truth. 

> We collect a dataset of human-written debates on hard reading comprehension questions where the judge has not read the source passage, only ever seeing expert arguments and short quotes selectively revealed by ‘expert’ debaters who have access to the passage. In our debates, one expert argues for the correct answer, and the other for an incorrect answer. Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy’s 74%. Debates are also more efficient, being 68% of the length of consultancies. 

> By comparing human to AI debaters, we find evidence that with more skilled (in this case, human) debaters, the performance of debate goes up but the performance of consultancy goes down. Our error analysis also supports this trend, with 46% of errors in human debate attributable to mistakes by the honest debater (which should go away with increased skill); whereas 52% of errors in human consultancy are due to debaters obfuscating the relevant evidence from the judge (which should become worse with increased skill). Overall, these results show that debate is a promising approach for supervising increasingly capable but potentially unreliable AI systems.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c44232ccf7a9d33700ce298ec56fe701c9d988b8fd2e5904.png)
# The dangers of reproducing while old

Excerpt:

I had my first child when I was 36 years old, which made me want to understand the risks of having children at different ages. Before looking into this, my impression was that the main biological problems with old-age parenthood had to do with not having the necessary health and vigor to care for young’uns, and I had heard that older women have trouble getting pregnant. While those are real issues, there are many others worthy of consideration.

My read of the evidence is that the risks of miscarriage and serious health problems for children, including autism and birth defects, increase significantly with parental (both paternal and maternal) age. The data I could find for most risks is not very fine-grained and not very precise, but I think this qualitative description matches the data: Risks start rising at around 30 years old for both mothers and fathers, rises gradually through about 35 for mothers and 40 for fathers, and then sharply after that.

[Read the full post here](https://www.garymm.org/blog/2023/11/10/the-dangers-of-reproducing-while-old/).
# Sam Altman fired from OpenAI

Basically just the title, see the OAI blog post for more details.

> Mr. Altman’s departure follows a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities. The board no longer has confidence in his ability to continue leading OpenAI.
> 
> In a statement, the board of directors said: “OpenAI was deliberately structured to advance our mission: to ensure that artificial general intelligence benefits all humanity. The board remains fully committed to serving this mission. We are grateful for Sam’s many contributions to the founding and growth of OpenAI. At the same time, we believe new leadership is necessary as we move forward. As the leader of the company’s research, product, and safety functions, Mira is exceptionally qualified to step into the role of interim CEO. We have the utmost confidence in her ability to lead OpenAI during this transition period.”

* * *

EDIT:

Also, Greg Brockman is stepping down from his board seat:

> As a part of this transition, Greg Brockman will be stepping down as chairman of the board and will remain in his role at the company, reporting to the CEO.

The remaining board members are:

> OpenAI chief scientist Ilya Sutskever, independent directors Quora CEO Adam D’Angelo, technology entrepreneur Tasha McCauley, and Georgetown Center for Security and Emerging Technology’s Helen Toner.

* * *

EDIT 2:

Sam Altman [tweeted the following](https://twitter.com/sama/status/1725631621511184771).

> i loved my time at openai. it was transformative for me personally, and hopefully the world a little bit. most of all i loved working with such talented people. 
> 
> will have more to say about what’s next later. 
> 
> ![🫡](https://abs-0.twimg.com/emoji/v2/svg/1fae1.svg) 

Greg Brockman [has also resigned](https://twitter.com/gdb/status/1725667410387378559).
# I think I'm just confused.  Once a model exists, how do you "red-team" it to see whether it's safe.  Isn't it already dangerous?

I think you get the point but say openAI "trains" GPT-5 and it turns out to be so dangerous that it can persuade anybody of anything and it wants to destroy the world.

We're already screwed, right?  Who cares if they decide not to release it to the public?  Or like they can't "RLHF" it now, right?  It's already existentially dangerous?

I guess maybe I just don't understand how it works.  So if they "train" GPT-5, does that mean they literally have no idea what it will say or be like until the day that the training is done?  And then they are like "Hey what's up?" and they find out?
# “Why can’t you just turn it off?”

If you're so worried about AI risk, **why don't you just turn off the AI** when you think it's about to do something dangerous?

On Friday, Members of the OpenAI board including Ilya Sutskever decided that they wanted to **"turn off"** OpenAI's rapid push towards smarter-than-human AI by firing CEO Sam Altman. 

The result seems to be that **the AI won**. The board has backed down after Altman rallied staff into a mass exodus. There's an implied promise of riches from the AI to those who develop it more quickly, and people care a lot about money and not much about small changes in x-risk. Of course this is a single example, but it is part of a pattern of people wanting to reap **localized** rewards from AI - recently the UK said it will [refrain from regulating AI 'in the short term'](https://www.ft.com/content/ecef269b-be57-4a52-8743-70da5b8d9a65), EU countries started lobbying to have [foundation models excluded from regulation](https://techcrunch.com/2023/11/16/mistral-eu-ai-act/).

That is why you cannot just turn it off. People won't *want* to turn it off[^1].


-----------------------------------------------------------

[^1]: There is a potential counterargument that once it becomes clear that AI is very dangerous, people will want to switch it off. But there is a conflicting constraint that it must also be possible to switch if off at that time. At early times, people may not take the threat seriously, and at late times they may take it seriously but not be able to switch it off because the AI is too powerful.
# Aaron Silverbook on anti-cavity bacteria

I recently released a podcast episode with Aaron Silverbook, a person within the LW-o-sphere, about his new start-up that produces a bacterium that might cure cavities, and also how cavities work and what's up with the bacteria we all have coating our teeth.

Here are links to [Lumina Probiotic (the brand name of this new type of bacterium)](https://www.luminaprobiotic.com/) and [Lantern Bioworks (the company making the bacterium)](https://www.lanternbioworks.com/).
# Emmett Shear to be interim CEO of OpenAI

*Update 2: this is kind of stale given more recent developments and I'm not planning to update it further; probably any further discussion should happen in* [*OpenAI: Facts from a Weekend*](https://www.lesswrong.com/posts/KXHMCH7wCxrvKsJyn/openai-facts-from-a-weekend) *or other newer posts.*

  
Update: un-paywalled article from the Verge: [https://www.theverge.com/2023/11/20/23967515/sam-altman-openai-board-fired-new-ceo](https://www.theverge.com/2023/11/20/23967515/sam-altman-openai-board-fired-new-ceo) 

  
This is still breaking as of 9:30 PT on Sunday, but according to a Bloomberg journalist:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/02fdc872018ecd7737156c13fff7e05121822ca725a294b7.png)

https://twitter.com/ashleevance/status/1726466910571610619

Paywalled article: [https://www.theinformation.com/articles/breaking-sam-altman-will-not-return-as-ceo-of-openai](https://www.theinformation.com/articles/breaking-sam-altman-will-not-return-as-ceo-of-openai)

> Emmett Shear, co-founder of Amazon-owned video streaming site Twitch, will take over as interim CEO, Sutskever said.

  
Manifold is not totally convinced: 

But I think this is mainly due to the permanent vs. interim issue.

More context: [https://twitter.com/ashleevance/status/1726469283734274338](https://twitter.com/ashleevance/status/1726469283734274338)

Emmett's twitter: [https://twitter.com/eshear](https://twitter.com/eshear)

Emmett's appearance on The Logan Bartlett show: [https://twitter.com/liron/status/1672986864297578501](https://twitter.com/liron/status/1672986864297578501)
# Sam Altman, Greg Brockman and others from OpenAI join Microsoft

That's very interesting.  
I think it's very good that board stood their ground, and maybe a good thing OpenAI can keep focusing on their charter and safe AI and keep commercialization in Microsoft.  
People that don't care about alignment can leave for the fat paycheck, while commited ones stay at OpenAI.  
What are your thought on implications of this for alignment?  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/740ef49ce02c3a31cdc704ca54e6468a57157c43d13eeb35.png)
# Short film adaptation of the essay "The Simple Truth" [eng sub]

I run a channel on YouTube. One of the directions is the shooting of short feature films about rationality. Previously, we shot a video based on Scott Alexander's essay "ARS LONGA, VITA BREVIS". This time we present to you an adaptation of "The Simple Truth" by Eliezer Yudkowsky. The video is in Russian, but we have added English subtitles using parts of the original text.

What essays would you be interested in seeing adapted in this way?

link to video:  
[https://youtu.be/3Ccj9G3jRm4?si=BhbGEMxQ1VdsPynA](https://youtu.be/3Ccj9G3jRm4?si=BhbGEMxQ1VdsPynA)

link to original essay:  
[https://www.lesswrong.com/posts/X3HpE8tMXz4m4w6Rz/the-simple-truth](https://www.lesswrong.com/posts/X3HpE8tMXz4m4w6Rz/the-simple-truth)
# How did you integrate voice-to-text AI into your workflow?

ChatGPT seems to have really awesome voice-to-text ability. However, it seems to only record within ChatGPT itself so can't be used to create notes or type in other programs and it's unclear to me how to best take advantage of the increased technological capabilities.

I'd love to hear about how people integrated the newest voice-to-text capabilities into their workflow.
# Vote on worthwhile OpenAI topics to discuss

I (Ben) recently made [a poll](https://www.lesswrong.com/posts/hc9nMipTXy2sm3tJb/vote-on-interesting-disagreements) for voting on interesting disagreements to be discussed on LessWrong. It generated a lot of good topic suggestions and data about what questions folks cared about and disagreed on. 

So, Jacob and I figured we'd try applying the same format to help people orient to the current OpenAI situation. 

What important questions would you want to see discussed and debated here in the coming days? Suggest and vote below.

How to use the poll
-------------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b75596d2bbed9af3be19585ef6fb2addea7ca816873cf5bb.png)

1.  **Reacts:** Click on the **agree/disagree reacts** to help people see how much disagreement there is on the topic.
2.  **Karma:** Upvote positions that **you'd like to read discussion** about.
3.  **New Poll Option:** Add new positions for people to take sides on. **Please add the agree/disagree reacts** to new poll options you make.

The goal is to show people where a lot of interest and disagreement lies. This can be used to find discussion and [dialogue](https://www.lesswrong.com/dialogues) topics in the future.
# Userscript to always show LW comments in context vs at the top

The out of context top-level display of comments when I navigate to them always bothered me, but up until recently I haven't realized there is a way to go to the actual comment via a simple URL change.   
  
From `https://www.lesswrong.com/posts/<postid>/?commentId=<commentid>`  
To `https://www.lesswrong.com/posts/<postid>#<commentid>`  
  
Here is a quick gpt4 generated userscript that does this. 

    // ==UserScript==
    // @name         Always show in-context comments
    // @namespace    http://tampermonkey.net/
    // @version      0.1
    // @description  Redirect from '?commentId=' format to '#'
    // @author       Vlad Sitalo
    // @match        https://www.lesswrong.com/*
    // @grant        none
    // ==/UserScript==
    
    
    (function() {
        'use strict';
    
        var checkAndRedirect = function() {
            var url = window.location.href;
            if(url.includes("?commentId=")){
                var newUrl = url.split("?commentId=").join("#");
                window.location.href = newUrl;
            }
        };
    
        // Run on page load
        checkAndRedirect();
    
        // Run on URL change
        var pushState = history.pushState;
        history.pushState = function() {
            pushState.apply(history, arguments);
            checkAndRedirect();
        };
    
        var replaceState = history.replaceState;
        history.replaceState = function() {
            replaceState.apply(history, arguments);
            checkAndRedirect();
        };
    
        window.addEventListener('popstate', checkAndRedirect);
    })();
# Portable Chargers are Great

Portable chargers are great and underrated. For [$20](https://www.amazon.com/Portable-Charger-26800mAh-Capacity-External/dp/B08729Z2JX) you can get a little box that can charge a phone ~5x or laptop ~1x. \[1\] You can pay a bit more for one with [built-in wall-charging and cables](https://www.amazon.com/VRURC-Portable-20000mAh-Charging-Replacement/dp/B0CDZNB954/). It's great throwing one in my backpack and knowing I [don't have to ration or worry](https://www.jefftk.com/p/rationing-spells-and-battery), especially when traveling. And if the kids tablets are running low, a friend needs a charge, or there's not a good place to plug my phone in overnight it's not an issue.

In general 27,000 mAh is a good upper size: past that and your airline might have limits (though the [FAA](https://www.faa.gov/hazmat/packsafe/lithium-batteries) allows airlines to permit up to 44,000 mAh). And once you're getting that big it probably makes more sense to get two for the flexibility and redundancy benefits.

I first got one in 2017, and it's still working well (though the USB-C port has gotten slightly loose):

[![](https://www.jefftk.com/portable-charger.jpg)](https://www.jefftk.com/portable-charger-big.jpg)

I've since gotten a couple more so the kids can have separate ones for their tablets on long trips.

  
\[1\] The math here is terrible: capacity should be measured in watt-hours, but instead everyone gives it in milliamp hours relative to some implied voltage. For USB power banks that's (always?) 3.6V, so 26,800 mAh is 96 Wh. My phone is [4,385 mAh](https://store.google.com/us/product/pixel_7a_specs) at 3.85V, so 17 Wh. My laptop is helpfully [spec'd in Wh](https://support.apple.com/kb/SP858): 99.6 Wh. Note that you do lose some capacity to conversion losses, so discount a bit.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02TPqYLHSR1vaKdiucf83amFqXosWbNe9du1Npn6AqDg9NpZp9pswii4AohKq2nR6gl), [mastodon](https://mastodon.mit.edu/@jefftk/111451918105265115)*
# [Bias] Restricting freedom is more harmful than it seems

- We do not censor other people more conventional-minded than ourselves. We only censor other people more-independent-minded than ourselves. Conventional-minded people censor independent-minded people. Independent-minded people do not censor conventional-minded people. [The most independent-minded people do not censor anyone at all.](https://www.youtube.com/watch?v=hmdTjDe5xKM&t=1s)
- Conventional-minded people cannot imagine ideas that are both heretical and true. Because of this, conventional-minded people cannot distinguish between heresy and falsity. To a conventional-minded person, "heretical" and "false" are the same thing. When conventional-minded people try to ban false statements, they actually ban heresy instead.

Any attempt to censor harmful ideas [actually suppresses the invention of new ideas (and correction of incorrect ideas)](http://paulgraham.com/orth.html) instead. [Freedom of speech](https://en.wikipedia.org/wiki/First_Amendment_to_the_United_States_Constitution) protects a society from its conventional-minded people accidentally suppressing truth in the name of safety.

This pattern doesn't just apply to freedom of speech. It applies to freedom[^1] of everything. Literally every regulation that restricts informed consenting adults from participating in mutually-beneficial activity $X$ ends up suppressing $X$ (usually in the name of safety).

[^1]: By this, I mean negative rights like freedom of speech, freedom of assembly, and freedom from unreasonable search and seizure. I do not include positive rights like "freedom from want" and healthcare. Negative rights protect weirdos from normies' biases. Positive rights contort the word "freedom". Positive rights demand government proaction, which (except for a handful of self-funded governments like Saudi Arabia's) is an implicit restriction of freedom.

You might ask yourself "Why would anyone ever want to do $X$?" and answer "Nobody." But that is because you're conventional-minded. If nobody wanted to do $X$ then there would be no need to restrict it. The more independent-minded you are, the more society's restrictions[^2] prevent you from creating value. The more conventional-minded you are, the more blind you are to the value that society's restrictions destroy.

[^2]: I originally wrote "laws" instead of "society's restrictions". There are many ways for for a society to restrict freedom other than via laws.

When conventional-minded people advocate for restrictions on freedom, they under-estimate the damage these restrictions cause. Most people are conventional-minded. Human societies systemically err on the side of too little freedom. Society is over-regulated[^3] in every domain of human experience, except those so new or so weird that the conventional-minded haven't noticed them (yet).

[^3]: Even domains with externalities tend to over-regulated because instead of imposing a tax (which allows some freedom while pricing in the externality), people tend to support prohibition instead.
# How "Pinky Promise" diplomacy once stopped a war in the Middle East

An incredible example of game theory's Generous Tit for Tat strategy being utilised in diplomacy during a real-world crisis - stopping a military escalation between Israel and Jordan in the eve of the 1990 Gulf War. Also includes some hilarious accounts from the King of Jordan himself (I can't figure out if he's aware of his comic timing or simply talking naturally).