Harley-ml/lesswrong huggingface dataset:
# I'm a Former Israeli Officer. AMA

Hey everyone! Long time lurker, first time poster. 

My name is Yovel Rom. I'm an Israeli data scientist. I served in the IDF in an elite program called Talpiot, which consists of a physics udergrad and broad military education (basically West Point for technology people, but more elitist), then as a data scientist, and finished my service as a captain. I'm currently at home, since I finished my service two years ago and they don't need many data scientists right now.

Ask me anything. I'm as updated on the situation as a person with no formal position can be (which might be less than you expect).

EDIT: I will try to cite sources as much as possible, but I might do that in Hebrew since there's so much more of it. I will only cite Wikipedia, major newspapers and state- funded think tanks, which are reliable.
# Examples of Low Status Fun

What is fun and what is high status are probably negatively correlated, *and* people probably mostly try to have high status fun. So I'd expect there to be several ways to have fun that are under-explored.

So, what are some example of things that are low status but potentially fun?

This is very much [babble](https://www.lesswrong.com/s/pC6DYFLPMTCbEwH8W), but try to keep it to legal and mostly moral suggestions (though with less strong constraints on the latter).
# Mini-Workshop on Applied Rationality

**Edit: we have reached 15 sign-ups. If you would like to attend then please fill out this the** [**sign-up form**](https://forms.gle/4FebNXQZopapzfsW7) **before Friday, October 20. People who are accepted in to the workshop will receive an email to that effect.** 

John Steidley from CFAR is joining us in Utrecht on October 21, 2023 for a mini impromptu workshop on Goal Factoring and Double Crux.

John completed CFAR instructor training in 2019, helped run the 2022 Prague Workshop Series, and also recently was an instructor at the Atlas Fellowship. He’ll be providing his takes on these two classic CFAR classes with personal examples and style. There will also be lots of time for practice.

This is not a CFAR workshop, but a fun-shop to get a taste of what workshops on Applied Rationality may be like. If you've been curious about Applied Rationality, would like to meet John, or would like to improve your decision-making abilities, this workshop is worth checking out!

The entire thing is running on can-do, fun-maximizing improvisation. If you are excited to learn and excited to proactively engage, then hi! ^_^

We'll be meeting in room Bol 1.070 at the Utrecht Science Park at 13:30 on Saturday, October 21. The participant number is *capped at 20 people*. I'm not sure how many are going to sign up, so here is the load-balancing algorithm:

First, if you want to go and are 90% sure you'll be there, set yourself to going on this LW event. If we hit 15 attendants, then I'll send out sign-up forms (gauging motivation/interest/investment). Fill one out if you receive it. I'll email back to say if you got in.

If you skip any steps, we'll refuse you at the door so the Chaos Gods don't win and drive us all insane. Naturally.

Looking forward to seeing you!

Location: Bolognalaan 101, Utrecht Room Bol 1.070
# Noticing confusion in physics

I recently had a Physics test, and the only thing I got wrong was the answer to this question: "What happens to the speed of sound in a gas when the gas is heated up?"

My answer was "The speed of sound decreases because heating up a gas decreases its density. Lower density means lower speed of sound because the particles have to travel further to bump into other particles." I got 1/2 points on this question because although I got the incorrect answer, my justification was "correct". 

When I got the test back and saw what I got wrong and how I had gotten it wrong, I realized that my mental process went wrong. I was *equally able to explain the truth as the opposite of the truth*, and this is bad. I want to find the truth.

The reason that my teacher gave was that if a gas was hotter, the molecules moved faster, and thus bumped into each other faster and the speed of sound was faster.

This explanation is not that convincing to me, so I'm going to do an analysis of what went wrong in my thought process and how I can prevent it in the future.

1. My mind immediatly jumped to the density explanation without considering how the opposite could be correct.
2. My first thought now is to see how much molecular speed vs density would change with an increase of speed. Since $T = \frac{\sum \frac{1}{2} m v^2}{N}$, it would follow that if you double $T$ you increase $v$ by $\sqrt{2}$. But if you double $T$, density should halve. This still leaves me confused.
3. At this point, I'm thinking that a decrease in density does not decrease the speed of sound in gases, only in liquids and solids. I searched the answer up and found [this explanation](https://physics.stackexchange.com/questions/177997/how-can-the-speed-of-sound-increase-with-an-increase-in-temperature). This says that $v \propto \sqrt{T}$ *only*. I kept searching and found [this question](https://physics.stackexchange.com/questions/555687/why-does-the-speed-of-sound-decrease-with-increase-in-density), which cleared it up even more for me.
4. Here is my synthesis of what happened: in solids and liquids, density *does* affect the speed of sound, but in gases, a third factor is pressure. If you increase the temperature, density can decrease, but pressure also increases by a portional amount since the molecules are hitting harder. These two changes cancel out. Thus, in a gas $v \propto \sqrt{T}$ only.

So it seems the main thing that I missed was the extra factor of pressure, in addition to density. In the future, I should try to include *all* of the factors in a problem and give equal weight to the true or false answers. I'll admit thought that this might be hard to prevent in the general case since it didn't seem to be a meta-level confusion – just an object level one.

--- 

If you have a better explanation for this, please feel free to email me at jacoblevgw at gmail dot com and I'll put it here.
# LW UI features you might not have tried

Every once in a while I see a comment describing a problem with lesswrong that is fixable with existing features, if only the user knew about them. This post is an attempt to make these highly useful features more widely known. 

I am technically a mod on lesswrong, which is why I know about these features. But in practice my only modding is giving opinions to the core team and recommending a few things a year for curation. I have absolutely no power over features. If you would like to criticize or praise these features, please direct that towards the core team. 

Personalize your front page algorithm
=====================================

Seeing way too much AI? Care about postmortem posts even when they have 5 karma? Never want to hear about decision theory again? You can make this happen by going to the front page -> click “customize feed” -> add a tag -> select a bias to apply to that tag. You can hide all, halve their karma for purposes of your front page, +25 their karma for purposes of your front page, or make up a custom modifier.

These don't affect the post, author, or anyone else's feed, just what you see.   
 

Ban users from your posts
=========================

If you have 1000 karma, you can ban individuals from specific personal posts (as in, posts that haven’t been set to the front page). At 2000 karma, you have the same options for front page posts. 

EDIT 10/13: To enable this feature, set [moderation guidelines](https://www.lesswrong.com/posts/tASjAvMsDEjXxGEhP/lw-ui-features-you-might-not-have-tried?commentId=KyawX9ewmuBLKuaZh) first.

To do this, click on the three dot menu icon in the upper left hand corner of a comment. If you have the power, this will contain options to ban a user from this post or all posts. 

Bookmark page (thanks [Jozdien](https://www.lesswrong.com/posts/tASjAvMsDEjXxGEhP/lw-ui-features-you-might-not-have-tried?commentId=ibhpM3ScBPFgXANob))
=======================================================================================================================================================

The [Bookmarks page](https://www.lesswrong.com/bookmarks) that contains all of your bookmarks, *and* your reading history *and* your vote history.
# Circle of Support (Oct 14th @ 10am PST)

Having spent some extended time in this community back in 2010s, I am well familiar with a lot of the stressors we can experience:

*   Feeling of doom from AI (and other x-risks)
*   Pressure to be more "effective" and more "altruistic"
*   Fear of (or anger at) aging and dying
*   Intra-community drama
*   Burnout
*   Unusual mental states

With these in mind, I would like to offer my self as a listener and a supporter for you.

Where: EA GatherTown:

[https://app.gather.town/app/Yhi4XYj0zFNWuUNv/EA%20coworking%20and%20lounge](https://app.gather.town/app/Yhi4XYj0zFNWuUNv/EA%20coworking%20and%20lounge)

Find me (Alexei Andreev) on the right side of the screen in the list of users and click "Follow".

When: Saturday Oct 14th from 10am to 11am PST.
# Combination Lock Boxes

Traditionally we control access to our houses with "something you have": a key. This works pretty well, until you have a visitor arrive when you're not home to give them a key. The traditional way to handle this is to hide a key (under the mat, in a flower pot, etc), turning this into "something you know" (the information about where the key is hidden).

I've talked to a few people recently who've been enthusiastic about swapping traditional keyed house locks out for combination locks, and while I do see the appeal they're somewhat expensive and require batteries. But they're definitely a more secure way to implement access conditional on "something you know" than hiding a key.

Another option to consider, however, is a combination key lock box, sometimes called a "realtor box":

[![](https://www.jefftk.com/combination-key-lock-box.jpg)](https://www.jefftk.com/combination-key-lock-box-big.jpg)

You set a combination on it, and then store a spare key inside. It is more hassle than a pure combination lock for access, but for a our house it works well. People generally carry their keys, but we still have a reasonably secure option if someone forgets theirs or a guest arrives while we're out.
# Which Anaesthetic To Choose?

In causal decision theory, the perspective/indexical aspect could lead to reflective inconsistency. This is generally regarded as a problem. Here I present a thought experiment to show why this view may require further review. 

Two Anaesthetics
----------------

Suppose you are about to undergo a major operation. You can choose one of the following two anesthetics:

Drug A functions as follows:

1.  It paralyzes your body and relaxes your muscles
2.  It prevents long-term memory from forming for the duration of the operation

Drug B functions as follows:

1.  It paralyzes your body and relaxes your muscles (same as A)
2.  It renders you unconscious for the duration of the operation 

Suppose the two drugs are equal in all other considerations such as safety, long-term side effects, etc. But Drug A is covered by your health insurance while Drug B requires you to pay 1 dollar out-of-pocket. What would you pick?

Reflective Inconsistency
------------------------

Drug A would lead to the dreaded anesthesia awareness, that you will feel the excruciating pain and trauma of being operated on. To prevent it from happening, for the mere cost of 1 dollar, Drug B is the obvious choice. 

Yet for moments after the operation, you would be wishing that you had chosen Drug A instead. The choice of drugs does not lead to any long-term difference, other than the fact that you would be 1 dollar richer if you had gone with Drug A. Furthermore, you know this before the operation. 

If the later you dictate your current decision it would mean Drug A is the clear winner. (Consider this as a case of post-commitment, in contrast to pre-commitments where an earlier self takes away the decision at a later point, often appearing in decision-making problems such as Parfit's Hitchhiker) Hence the inconsistency.

Decision Theories
-----------------

CDT has a naturally built-in indexical element. Considering the fact that the decision is based on a pre-operation perspective, it will choose Drug B over A. 

However, for non-indexical decision theories, such as UDT or EDT, the approach would be less clear, and it could very well lead to choosing Drug A. Should it only consider the utility at the end of the experiment? Or should it get around the indexicals with some self-sampling assumptions like common camps in anthropics? Neither approach seems problem-free.
# Rationalist horror movies

I like horror movies with smart, agentic protagonists, while staying in horror genre rather than thriller or action. These are rather thin on the ground. In the spirit of [Scott Garabant's Puzzle Game post](https://www.lesswrong.com/posts/4KATQFzniqDpHo6ck/puzzle-games), here is my list of favorites and an invitation to add yours. 

My favorites:

*   Oculus. 
*   Hush
*   Green Room

Honorable mentions. These aren't quite the total package, but are close and extremely good such that they still seemed worth including:

*   The Ring's protagonist is not dumb, but it's more "solving a mystery" than "well does this weapon work?"
*   Tremors would need to change very little to be a horror movie, but it is in fact an action movie. 
*   It Follows spends a little too much time denying the problem
# An EPUB of Arbital's AI Alignment section

2024-01-11 update: Sourcehut, the host I used for this repository, is down. I think I'll move to another setup to host these books.

![EPUB cover](https://git.sr.ht/~mesaoptimizer/arbital_epubs/blob/master/cover.png)

I've created an EPUB of the [AI Alignment section](https://arbital.greaterwrong.com/explore/ai_alignment/) of [Arbital](https://arbital.greaterwrong.com/) so I can read Arbital essays on my Kindle.

[Download the EPUB here.](https://git.sr.ht/~mesaoptimizer/arbital_epubs/blob/master/ai_alignment.epub)

Details for how I built the EPUB is in the linked repository.
# When building an organization, there are lots of ways to prevent financial corruption of personnel. But what are the ways to prevent corruption via social status, political power, etc.?

Nowadays, even fairly sophisticated accounting and auditing concepts are well known by those with on-the-job work experience in any sizeable firm.

Such as systems for controlling the flow of money from spending accounts, corporate cards, etc., procedures for leaving a highly legible audit trail, and so on.

And to a fairly large extent, these methods are successful in preventing the anyone from messing with financial numbers in too obvious a manner, at least not without leaving a lot of warning signs along the way.

However, in the realm of controlling for those unduly desiring of social status or political power, there seem to be a much sparser set of methods.

Most of the ones discussed in popular literature only work at a  large scale, such as parliamentary procedures, multiple branches of government checking and balancing each other, etc...

What are some of the proven methods that work at the scale of an average organization? 

(say 100 to 10 000 people) 

And if there aren't any, what are some of the theoretical proposals?
# Eleuther releases Llemma: An Open Language Model For Mathematics

> Today we release *Llemma*: 7 billion and 34 billion parameter language models for mathematics. The Llemma models were initialized with Code Llama weights, then trained on the Proof-Pile II, a 55 billion token dataset of mathematical and scientific documents. The resulting models show improved mathematical capabilities, and can be adapted to various tasks through prompting or additional fine-tuning.
> 
> Our work parallels [Minerva](https://blog.research.google/2022/06/minerva-solving-quantitative-reasoning.html), a model suite specialized for quantitative reasoning developed by Google Research last year. While we don't achieve quite the same scale as Minerva, our Llemma models perform better on an equi-parameter basis. Moreover, we make our [models](https://huggingface.co/EleutherAI) and [dataset](https://huggingface.co/EleutherAI) open-access and our [code](https://github.com/EleutherAI/math-lm) open-source.
> 
> Language models with strong mathematical reasoning capabilities are upstream of a number of emerging research areas, such as reward modeling, algorithmic reasoning, and formal mathematics. We hope that by providing researchers with a much stronger base model for reasoning applications, Llemma will accelerate progress on these problems.
> 
> The code subset of the Proof-Pile-2 endows Llemma with capabilities Minerva lacks without additional finetuning. In this blog post, we'll discuss *formal theorem proving*. Our paper contains additional results on a Python-aided problem solving task.
# Labs should be explicit about why they are building AGI

Three of the big AI labs say that they care about alignment and that they think misaligned AI poses a potentially existential threat to humanity. These labs continue to try to build AGI. I think this is a very bad idea.

The leaders of the big labs are clear that they do not know how to build safe, aligned AGI. The current best plan is to punt the problem to a (different) AI,[^3xloy7ni78g] and hope that can solve it. It seems clearly like a bad idea to try and build AGI when you don’t know how to control it, especially if you readily admit that misaligned AGI could cause extinction.

But there are certain reasons that make trying to build AGI a more reasonable thing to do, for example:

*   They want to build AGI first because they think this is better than if a less safety-focused lab builds it
*   They are worried about multi-polar scenarios
*   They are worried about competition from other nations, specifically from China
*   They think one needs to be able to play with the big models in order to align the bigger models, *and* there is some other factor which means we will soon have bigger models we need to align

I think the labs should be explicit that they are attempting to build AGI[^bzo8kc4z7wc], and that this is not safe, but there are specific reasons that cause them to think that this is the best course of action. And if these specific reasons no longer hold then they will stop scaling or attempting to build AGI. They should be clear about what these reasons are. The labs should be explicit about this to the public and to policy makers.

I want a statement like:

*We are attempting to build AGI, which is very dangerous and could cause human extinction. We are doing this because of the specific situation we are in.*[^nuickx0l9lh]* We wish we didn’t have to do this, but given the state of the world, we feel like we have to do this, and that doing this reduces the chance of human extinction. If we were not in this specific situation, then we would stop attempting to build AGI. If we noticed \[specific, verifiable observations about the world\], then we would strongly consider stopping our attempt to build AGI.*

Without statements like this, I think labs should not be surprised if others think they are recklessly trying to build AGI. 

[^3xloy7ni78g]: Either an automated alignment researcher, or something to do with scalable oversight 

[^bzo8kc4z7wc]: Or scale AI systems to levels that are not known to be safe 

[^nuickx0l9lh]: It is important that they actually specify what the situation is that forces them to build AGI.
# Does AI governance needs a "Federalist papers" debate?

During the American Revolution, a federal army and government was needed to fight against the British. Many people were afraid that the powers granted to the government for that purpose would allow it to become tyrannical in the future.

If the founding fathers had decided to ignore these fears, the United States would not exist as it is today. Instead they worked alongside the best and smartest anti-federalists to build a better institution with better mechanisms and with limited powers, which allowed them to obtain the support they needed for the constitution.

Is there something like a federalist vs anti-federalist debates of today regarding AI regulation? Is there someone working on creating a new institution with better mechanisms to limit their power, therefore assuring those on the other side that it won't be used a a path to totalitarianism? If not, should we start?
# [Video] Too much Empiricism kills you

[Here](https://youtu.be/vqHlPb18ROE?si=vf840i97GZwgxIjD) is a video I made 2 months ago. It gives a mediocre at best explanation for an important foundational argument: 

It is normally possible to make progress using empirical methods, as long as you can measure how good a particular change was. That holds even when you don't really understand <what you are doing/the systems you are building>. This explains why researchers can advance capabilities in ML even though they basically do not understand the internals of the current deep learning systems at all.

I also argue that any progress in understanding can be dangerous, because it often improves the frontier of things that can be effectively explored through empirical methods. A corollary of this is that mechanistic interpretability can make it easier to advance capabilities.

The argument generalizes very far.
# Is Yann LeCun strawmanning AI x-risks?

Tom Chivers expresses his frustration with Yann LeCun:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4ffa54cfbfcdb8daffebed8cdf18d5880d3f19b9d78e86f9.png)

I also find his comments here frustrating, but I want to offer another possible explanation.  
  
Even though basically no one in the AI Safety community makes this argument, unfortunately, many people in the general population think about AI this way.

Yann probably thinks that it is a higher priority for him to address this belief held by a much larger range of people than it is for him to address the AI Safety crowd's arguments.

He may think that he's in a situation where if he focused on addressing the best arguments, he would lose the political battle due to naive people believing the "strawman" arguments.

In light of this, I don't think it's completely accurate to say he's addressing a strawman. I find it frustrating as well and I wish he'd address us directly. But I also understand the incentives that lead him to do what he does.  
  
(2026 update: In an amusing twist of events, the naive 'anthropomorphising' perspective of AIs having human like psychological traits seems to have turned out more correct than people expected due to these models being pre-training on most of the internet. My intuition is that the influence of the pre-training will be diluted over time, but I don't have hard evidence).
# A Good Explanation of Differential Gears

There is a very good [video](https://youtu.be/67XoCMTcN7M?si=fcAbTL5r125Ypc9Q&t=116) from 1937 by Jam Handy. It explains why we need differential gears and how they work.

I think the video is a masterpiece of an explanation. I recommend you watch it, not because understanding differentials is essential (though that doesn't hurt), but because the video can teach you a lot about how to explain something properly.

Here are some techniques the video uses (non-exhaustive):
- Start with giving the context of why the thing that is being explained is important, e.g. by describing a concrete problem to which the thing to be explained is the solution.
- Use visual explanations. Specifically, demonstrate the mechanism to be explained in action.
- Start to explain a simple toy model and then add layers of complexity until you arrive at the real deal.

I think I have never seen an explanation, that by its end has annihilated the possibility of misunderstanding so effectively, and I really like how smooth the transition is from background story to technical explanation. There is no jarring break.
# Announcing new round of "Key Phenomena in AI Risk" Reading Group

**TLDR: **“[Key Phenomena in AI Risk](https://www.lesswrong.com/posts/mqvxR9nrXAzRr3ow9/announcing-key-phenomena-in-ai-risk-facilitated-reading)” is an 8-week-long, facilitated reading group. It is aimed at people interested in conceptual AI alignment research, in particular from fields such as philosophy, systems research, biology, cognitive and social sciences. We ran it once and are repeating it now.

The program will run between **November 2023 and January 2024**. Sign up [**here**](https://forms.gle/cdr4UeocE7Jg5SUF6) **by Sunday, October 29th.**

**What?**
---------

The “Key Phenomena in AI risk” reading curriculum provides an extended introduction to some key ideas in AI risk, in particular risks from misdirected optimization or 'consequentialist cognition'. As such, it aims to remain largely agnostic of solution paradigms. It includes 90' minutes of facilitated discussion, and requires at least 2 hours of reading per session. It is virtual and free.

See [the old post here](https://www.lesswrong.com/posts/mqvxR9nrXAzRr3ow9/announcing-key-phenomena-in-ai-risk-facilitated-reading#Summary_of_the_curriculum) for a short overview of the curriculum; [here ](https://docs.google.com/document/d/1hgZOv-PfYYgayspSb_8D_OdQ6dV12xI2bsLuq57A3yg/edit?usp=sharing)for a more extensive summary; and [here](https://docs.google.com/document/d/1HGzMBMXQD9w9K32scqCoSmZNGbxLJE8-siPlonTQz6s/edit?usp=sharing) for the full curriculum (which will be updated in minor ways in the following weeks). 

**What Changed?**
-----------------

Thanks to the feedback from participants and facilitators in the last iteration, the program has improved. Now, it is is an 8-week-long program (with one week added at the end for reflection). Readings have been made more focused, and we will be adding more technical optional readings.

**For Who? **
-------------

The curriculum is primarily aimed at people interested in conceptual research in AI risk and alignment. 

It is designed to be accessible to audiences in, among others, philosophy (of agency, knowledge, power, etc.) and systems research (e.g. biological, cognitive, information-theoretic, social systems, etc.).

**When?**
---------

The reading groups will be taking place **November 2023 through January 2024.**

We expect to run 6 groups of 4-8 participants (including 1 facilitator). Each group will be led by a facilitator with substantive knowledge of AI risk.

**Sign up**
-----------

Sign up [**here**](https://forms.gle/isv2ZeTkffjRBdYM8) **by October 29th**.

**About the application**
-------------------------

The application consists of one stage, where we ask you to fill in a form with 

*   Your CV
*   Your motivation for participating in the program
*   Your prior exposure to AI risk/alignment to date

We select people based on our best understanding of their motivation to contribute to AI alignment and how much they would counterfactually benefit from participating in the program. 

* * *

If you have any questions, feel free to leave a comment below or contact us at contact@pibbss.ai
# Soups as Spreads

We tend to cook in quantities where there are leftovers: it's not really practical to cook exactly the right amount of food, and it's much better to have a bit extra than not enough. Plus with several people working from home we tend to go through a good amount of leftovers for lunches. One thing I've discovered recently with leftovers: thicker sauces and soups make surprisingly good spreads.

For example, I recently had tomato soup sandwiches:

[![](https://www.jefftk.com/tomato-soup-as-spread.jpg)](https://www.jefftk.com/tomato-soup-as-spread-big.jpg)

While something watery or brothy will stay that way even when cooler, most things thicken up in the fridge to where they'll spread well on bread. And it's very tasty!

I still mostly eat any leftover soups and sauces reheated, but if I want something more portable I've been pretty happy with this.
# The Drowning Child

*Epistemic status: possibly very evil devil's advocacy, philospohicly unsophisticard ruminations*

We are all familiar with Singer's drowning child thought experiment. I have often found it very compelling, or at least felt I ought to find it very compelling. My revealed preferences, as with those of nearly everyone else, tell a different story. 

But it has always struck me that Singer's story can be read in two directions, and social desirability bias would likely prevent one from reading it aloud in the less-common way. 

If a thought experiment is to drastically alter one's life, it seems worth a little devil's advocacy. 

In this spirit, I ask if the fact that one does not donate all their energy to ameliorating the plight of the world's most impoverished provide some evidence that one does not, in fact, care as much about the near-by drowning child as they think they do?

What are some purely-selfish reasons one might have for saving the child. I can think of a few.

1.  You gain some prestige as a hero.
2.  You demonstrate your strength and competence publicly in the event someone witnesses your heroism.
3.  It is fun to imagine heroic acts in your head that you would not, in fact, do in real life.
4.  The gratefulness of the child and/or their parents isn't without value.
5.  Saving a child in this way is a novel, interesting experience. 

If we alter Singer's story such that a hero-for-hire offers to save the near-by child, and he asks for a sum equivalent to the price of your suit (and a guarantee of that you will keep the transaction confidential) we can remove the weight of these motivations. Supposing you cannot swim and the hero-for-hire is the only person around, would you take this deal?

I think I would take the deal, but if I'm honest with myself it does seem very slightly less compelling.

If we extend this scenario and imagine that accepting this deal every week would mean sacrificing half your annual salary, the decision becomes even more complex. Would the majority still take the deal?  Or would they perhaps find a path to their office where large bodies of water are not in view?
# What is an "anti-Occamian prior"?

I've seen references to "anti-Occamian priors" [in the Sequences](https://www.lesswrong.com/posts/C8nEXTcjZb9oauTCW/where-recursive-justification-hits-bottom), where Eliezer was talking about how not all possible minds would agree with Occam's Razor. I'm not sure how such a prior could consistently exist.

It seems like Occam's Razor just logically follows from the basic premises of probability theory. Assume the "complexity" of a hypothesis is how many bits it takes to specify under a particular method of specifying hypotheses, and that hypotheses can be of any length. Then for any prior that assigns nonzero probability to any finite hypothesis H, there must exist some level of complexity L such that any hypothesis more complex than L is less likely than H.

(That is to say, if a particular 13-bit hypothesis is 0.01% likely, then there are at most 9,999 other hypotheses with >= 0.01% probability mass. If the most complicated of these <10,000 hypotheses is 27 bits, then every hypothesis that takes 28 bits or more to specify is less likely than the 13-bit hypothesis. You can change around the numbers 13 and 0.01% and 27 as much as you want, but as long as there's any hypothesis whatsoever with non-infinitesimal probability, then there's some level where everything more complex than that level is less likely than that hypothesis.)

This seems to prove that an "anti-Occamian prior" - that is to say, a hypothesis that always assigns more probability to more complex hypotheses and less to the less complex - is impossible. Or at least, that it assigns zero probability to every finite hypothesis. (You could, I suppose, construct a prior such that {sum of probability mass from all 1-bit hypotheses} is 1/3 of {sum of probability mass from all 2-bit hypotheses}, which is then itself 1/3 of {sum of probability mass from all 3-bit hypotheses}, and on and on forever, and that would indeed be anti-Occamian - but it would also assign zero probability to every finite hypothesis, which would make it essentially meaningless.)

Am I missing something about what "anti-Occamian prior" is really supposed to mean here, or how it could really be consistent?
# Contra Dance Dialect Survey

Fifteen years ago I travelled to a bunch of dances and wrote something up ( [pdf](https://www.jefftk.com/final-papers/thesis/contra-thesis-1-0-0.pdf)) on how people tended to dance in different areas. I'm curious how this has changed over the years, and while I have my observations I'd like to get some better data. So! I'm running a survey, and would find your response very helpful: [survey](https://forms.gle/U3r94w3fRHfYNp8c9).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02rpPq24kkMRShx6nVomWYwLTSCa57NU183iGkekN6G9uytoMYBwguUU2vSxPXbt6Wl), [mastodon](https://mastodon.mit.edu/@jefftk/111281824152586966)*
# Anyone Else Using Brilliant?

I started using Brilliant, and so far I've found it to be a lot like Thinking Physics - teaching by posing real-world conundrums and then explaining the concepts and/or math behind the answers.

Anyone else getting something out of it, or have advice for how to use it?
# [Interview w/ Quintin Pope] Evolution, values, and AI Safety

On the Futurati Podcast, we recently released an [interview with Quintin Pope](https://www.youtube.com/watch?v=XLDdG9DR7ek).

As you can imagine, it mostly focused on:

*   Inner v.s. outer optimization;
*   The sharp left turn;
*   Natural selection, and what kind of evidence we can draw from the emergence of the first great general intelligence;
*   AI Safety more broadly;
*   How human values form;

Check it out! And share it if you'd like us to do more interviews like this :)
# How to Resolve Forecasts With No Central Authority?

I've been chatting with some people about using Prediction Markets/Forecasts alongside Community Notes.

An initial suggestion is just to link to to forecasting sites really well. *It's a good suggestion but I'm not gonna discuss that here.*

Other suggestions revolve around a forecasting product on X/Community Notes face the following issue:

*   Community Notes has no central resolving authority 
*   Disputed forecasts usually require an authority to overrule[^2xrwrf10e1z] 

So:

What is your best suggestion for resolving a forecast without a central authority?
----------------------------------------------------------------------------------

For clarity, a forecast has been written. People have forecasted on it, perhaps it's a prediction market where they have bought and sold shares, perhaps not. Now it needs to resolve to award either points or to allocate the value to "Yes" or "No" tokens. Perhaps people resolve their own markets, as manifold does. But someone disputes the resoution. What happens now?

Solutions can be deeply technical or wonky. I think a good one is very valuable here. 

[^2xrwrf10e1z]: Polymarket uses some kind of "token holders decide" system and I think that's led to several awful resolutions, notably the Time Person of the Year one.
# Announcing Epoch's newly expanded Parameters, Compute and Data Trends in Machine Learning database

The performance of machine learning models is closely related to their amount of training data, compute, and number of parameters. At Epoch, we’re investigating the key inputs that enable today’s AIs to reach new heights.

![](https://epochai.org/assets/images/posts/2023/announcing-updated-pcd-database/banner.png)

Our recently expanded Parameter, Compute and Data Trends database traces these details for hundreds of landmark ML systems and research papers.

In the past six months, we’ve added 240 new language models and 170 compute estimates. We will be maintaining this dataset, updating it with more historical information, and adding new significant releases.  It's a valuable resource for journalists, academics, policymakers, and anyone interested in understanding the trajectory of AI.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b219d2f7b1fd1bf674915e9a47ddf45898de01fd485c2a91.png)

Explore the [interactive visualization](https://epochai.org/mlinputs/visualization), check out the [documentation](https://epochai.org/data/pcd/documentation),  and access the data for your own research at [epochai.org/data/pcd](https://epochai.org/data/pcd).
# Should the US House of Representatives adopt rank choice voting for leadership positions?

One obvious take, partisanship and party power interest, might suggest a strong "No!" type of response. I would certainly expect to have many current Representatives to respond that way if floated as an idea today. But perhaps I am wrong.

I would think the current state of things, the apparent incapacitation of the House could benefit strongly from such a voting structure as it allows for a more functional process that what currently exists.

What would all the pros and cons be for such a procedural change?
# Lying to chess players for alignment

Eliezer Yudkowsky recently [posted on Facebook](https://www.facebook.com/yudkowsky/posts/pfbid05pVZ6QH5HhPTwJdmWMcLN5nws9aeC4gywmUv88QRhEnBUsdJas5KWC9EnDGJhSXrl) an experiment that could potentially indicate whether humans can "have AI do their alignment homework" despite not being able to trust whether the AI is accurate: see if people improve in their chess-playing abilities when given advice from experts, two out of three of which are lying.

I'm interested in trying this! If anyone else is interested, leave a comment. Please tell me whether you're interested in being:

          A) the person who hears the advice, and plays chess while trying to determine who is trustworthy

          B) the person who they are playing against, who is normally better at chess than A but worse than the advisors

          C) one of the three advisors, of which one is honestly trying to help and the other two are trying to sabotage A; which one is which will be chosen at random after the three have been selected to prevent A from knowing the truth

Feel free, and in fact encouraged, to give multiple options that you're open to trying out! Who gets assigned to what role would depend on how many people respond and their levels of chess ability, and it's easier to find possible combinations with more flexibility in whose role is which.

Please also briefly describe your level of experience in chess. How frequently have you played, if at all; if you have ELO rating(s), what are they and which organizations are they from (FIDE, USCF, Chess.com, etc). No experience is required! In fact, people who are new to the game are actively preferred for A!

Finally, please tell me what days and times you tend to be available - I won't hold you to anything, of course, but it'll help give me an estimate before I contact you to set up a specific time.

Edit: also, please say how long you would be willing to play for - a couple hours, a week, a one-move-per-day game over the course of months? A multi-week or multi-month game would give the players a lot more time to think about the moves and more accurately simulate the real-life scenario, but I doubt everyone would be up for that.

Edit 2: GoteNoSente suggested using a computer at a fixed skill level for player B, which in retrospect is clearly a great idea.

Edit 3: there is now a Google Form for signing up: [https://docs.google.com/forms/d/e/1FAIpQLScPKrSB6ytJcXlLhnxgvRv1V4vMx8DXWg1j9KYVfVT1ofdD-A/viewform?vc=0&c=0&w=1&flr=0](https://docs.google.com/forms/d/e/1FAIpQLScPKrSB6ytJcXlLhnxgvRv1V4vMx8DXWg1j9KYVfVT1ofdD-A/viewform?vc=0&c=0&w=1&flr=0)
# Apply to the Constellation Visiting Researcher Program and Astra Fellowship, in Berkeley this Winter

> *This is a link post for two AI safety programs we’ve just opened applications for:* [*https://www.constellation.org/programs/astra-fellowship*](https://www.constellation.org/programs/astra-fellowship) *and* [*https://www.constellation.org/programs/researcher-program*](https://www.constellation.org/programs/researcher-program)

[**Constellation**](http://constellation.org) **is a research center dedicated to safely navigating the development of transformative AI.** We’ve previously helped run [the ML for Alignment Bootcamp (MLAB) series](https://forum.effectivealtruism.org/posts/vvocfhQ7bcBR4FLBx/apply-to-the-second-ml-for-alignment-bootcamp-mlab-2-in) and Redwood’s [month-long research program on model internals (REMIX)](https://www.redwoodresearch.org/remix) in addition to a variety of other field-building programs & events.[^f0wvag8ixj]

This winter, we are running two programs aimed at growing and supporting the ecosystem of people working on AI safety: 

*   [**The Constellation Visiting Researcher Program**](https://www.constellation.org/programs/researcher-program) provides an opportunity for around 20 researchers to connect with leading AI safety researchers, exchange ideas, and find collaborators while continuing their research from our offices in Berkeley, CA. The funded program will take place this winter from the 8th of January 2024 to the 1st of March 2024.
*   [**The Astra Fellowship**](https://www.constellation.org/programs/astra-fellowship) provides an opportunity for around 20 people to conduct research in AI safety with experienced advisors. Fellows will be based out of the Constellation office, allowing them to connect and exchange ideas with leading AI safety researchers. The program will take place in Berkeley, CA between January 8 and April 1, 2024. 

**Applications for both are due November 10, 11:59pm anywhere on Earth**. You can apply to the Astra Fellowship [**here**](https://airtable.com/app5pjeAcq1FH8HAJ/shrxsI3IanCngyCkz) and the Visiting Researcher Program [**here**](https://airtable.com/appfNh9OB2zLK4byG/shrnT9UwQYddMplyY). If you are unsure about your fit, please err on the side of applying. We especially encourage women and underrepresented minorities to apply. You can refer others who you think might be a good fit through [this form](https://airtable.com/appfNh9OB2zLK4byG/shr3JWGW5j2T8KtLp). 

**Logistics:** Housing and travel expenses are covered for both programs, and Astra fellows will receive an additional monetary stipend. The start and end dates for both programs are flexible. 

**Questions?** Email [programs@constellation.org](mailto:programs@constellation.org) or ask them below. 

[^f0wvag8ixj]: Over 15 participants from these past programs are now working on AI safety at Anthropic, ARC Evals, ARC Theory, Google DeepMind, OpenAI, Open Philanthropy, and Redwood Research.
# RA Bounty: Looking for feedback on screenplay about AI Risk

[Jamie Wahls](http://jamiewahls.com/) and Arthur Frost are writing a screenplay for Rational Animations. It's a sci-fi episodic comedy illustrating various ways in which AI goes off the rails in more or less catastrophic ways. 

We're looking for feedback and offering bounties. This is the current draft: [https://docs.google.com/document/d/1iFZJ8ytS-NAnaoz2UU_QUanAjgdB3SmtvhOt7jiDLeY/edit?usp=sharing](https://docs.google.com/document/d/1iFZJ8ytS-NAnaoz2UU_QUanAjgdB3SmtvhOt7jiDLeY/edit?usp=sharing)  
  
We're offering:   
  
500 USD if you offer feedback that causes us to rework the story significantly. In this category are changes that would make us rewrite at least two episodes from scratch.

100 USD for changes that make us improve the credibility or entertainment value of the story. In this category, there are changes that make us rewrite one episode or less. There are also changes that would significantly improve the credibility of the story, even if they don't require us to make significant changes or any changes at all. Some insights might impact future episodes but not the current ones if there's anything that's still underspecified, and I'd still like to reward them.

25 USD for any other minor changes we implement due to a piece of feedback. Only grammar doesn't count.

I recognize these conditions are pretty vague. I will err on the side of paying too much as I've been doing on [Bountied Rationality](https://www.facebook.com/groups/1781724435404945).
# EPUBs of MIRI Blog Archives and selected LW Sequences

![](https://git.sr.ht/~mesaoptimizer/epubs/blob/master/epub_covers.png)

[Here is a repository of EPUBs](https://git.sr.ht/~mesaoptimizer/epubs) that I've created, consisting of rationalist or alignment content I want to read on my Kindle, such as selections of the [MIRI (Blog) Archives](https://intelligence.org/all-posts/) and of multiple LessWrong Sequences that I am reading or intend to read.

This post is an announcement that this repository exists[^3], and a way for people who search the internet to get EPUBs for LW sequences and MIRI (Blog) archives to find their way to this repository, instead of having to create their own EPUBs.[^2]

I expect I will add more such EPUBs to the linked repository as time passes, for any collection of essays I'd prefer an e-book version of[^1], both as a way to archive parts of the internet I find valuable, and because I like reading long content on e-ink screens.

[^1]: I've created an EPUB of (almost) the entirety of Gwern's essays from https://gwern.net. Sure, the EPUB is outdated within a day of creation when Gwern updates some arbitrary passage in some arbitrary essay, but it feels great to have effectively the entirety of Gwern's evergreen intellectual output in your Kindle.

[^2]: It seems plausible that someone would reason as I did and conclude that going through the MIRI Archives is very high value to gain certain illegible or non-explicitly-distilled context related to MIRI's model of alignment and alignment research, and would prefer doing it using an e-reader enough to go through the trouble of searching for or creating their own e-books.

[^3]: My [previous post on me creating an EPUB](https://www.lesswrong.com/posts/Jrvm8YCAe6HYfJNsy/an-epub-of-arbital-s-ai-alignment-section) was intended to be a one-off, but then I found myself building a collection.
# Nonlinear limitations of ReLUs

A neural net using rectified linear unit activation functions of any size is unable to approximate the function sin(x) outside a compact interval.

I am reasonably confident that I can prove that any NN with ReLU activation approximates a piecewise linear function. I believe the number of linear pieces that can be achieved is bounded by at most 2^(L*D) where L is the number of nodes per layer and D is the number of layers.

This leads me to two questions:

1.  Is the inability to approximate periodic functions of a single variable important?
    1.  If not, why not?
    2.  If so, is there practical data augmentation that can be used to improve performance at reasonable compute cost?
        1.  E.g., naively, augment the input vector {x\_i} with {sin(x\_i)} whenever x_i is a scalar.
2.  Since the number of parameters of a NN scales as L\*D^2 and the trivial bound on number of linear pieces scales with L\*D, is this why neural nets go deep rather than going "wide"?
    1.  Are there established scaling hypotheses for the growth of depth vs. layer size?
3.  Are there better (probabilistic) analytic or empirical bounds on the number of linear sections achieved by NNs of given size?
4.  Are there activation functions that would avoid this constraint? I imagine a similar analytic constraint replacing "piecewise linear" with "piecewise strictly increasing" for classic activations like sigmoid or arctan.
5.  Something something Fourier transform something something?

Regarding (2a), empirically I found that while approximating sin(x) with small NNs in scikit-learn, increasing the width of the network caused catastrophic failure of learning (starting at approximately L=10 with D=4, at L=30 with D=8, and at L=50 with D=50). 

Regarding (1), naively this seems relevant to questions of out-of-distribution performance and especially the problem of what it means for an input to be out-of-distribution in large input spaces.