Harley-ml/lesswrong huggingface dataset:
# Michael Dickens' Caffeine Tolerance Research

[Michael Dickens](https://mdickens.me/) has [read](https://mdickens.me/2024/03/02/caffeine_tolerance/) [the research](https://mdickens.me/2024/03/29/does_caffeine_stop_working/) and performed [two](https://mdickens.me/2024/04/11/caffeine_self_experiment/) [self-experiments](https://mdickens.me/2024/06/24/continuing_caffeine_self_experiment/) on whether consuming caffeine builds up tolerance, and if yes, how quickly.

[First literature review](https://mdickens.me/2024/03/02/caffeine_tolerance/):

> What if instead of taking caffeine every day, you only take it intermittently—say, once every 3 days? How often can most people take caffeine without developing a tolerance?
>
> The scientific literature on this question is sparse. Here’s what I found:
>
> 1. Experiments on rats found that rats who took caffeine every other day did not develop a tolerance. There are no experiments on humans. There are no experiments that use other intermittent dosing frequencies (such as once every 3 days).
> 2. Internet forum users report that they can take caffeine on average once every 3 days without developing a tolerance. But there’s a lot of variation between individuals.

[Second literature review](https://mdickens.me/2024/03/29/does_caffeine_stop_working/):

> If you take caffeine every day, does it stop working? If it keeps working, how much of its effect does it retain?
> 
> There are many studies on this question, but most of them have severe methodological limitations. I read all the good studies (on humans) I could find. Here’s my interpretation of the literature:
>
> * Caffeine almost certainly loses some but not all of its effect when you take it every day.
> * In expectation, caffeine retains 1/2 of its benefit, but this figure has a wide credence interval.
> * The studies on cognitive benefits all have some methodological issues so they might not generalize.
> * There are two studies on exercise benefits with strong methodology, but they have small sample sizes.

First experiment:

> I conducted an experiment on myself to see if I would develop a tolerance to caffeine from taking it three days a week. The results suggest that I didn’t. Caffeine had just as big an effect at the end of my four-week trial as it did at the beginning.
>
> This outcome is statistically significant (p = 0.016), but the data show a weird pattern: caffeine’s effectiveness went up over time instead of staying flat. I don’t know how to explain that, which makes me suspicious of the experiment’s findings.

Second experiment:

> This time I tested if I could have caffeine 4 days a week without getting habituated.
>
> Last time, when I took caffeine 3 days a week, I didn’t get habituated but the results were weird. This time, with the more frequent dose, I still didn’t get habituated, and the results were weird again!
> […]
> But it looks like I didn’t get habituated when taking caffeine 4 days a week—or, at least, not to a detectable degree. So I’m going to keep taking caffeine 4 days a week.
> ### When I take caffeine 3 days in a row, do I habituate by the 3rd day?

> The evidence suggests that I don’t, but the evidence is weak.
# Why Reflective Stability is Important

Imagine you have the optimal AGI source code $O$. If you run $O$ the resulting AGI process will not self-modify.[^42imbsrhqz4] You get a non-self-modifying AI for free.

This would be good because a non-self-modifying AI is much easier to reason about. 

Imagine some non-optimal AGI source code $S$. Somewhere in $S$, there is a for loop that performs some form of program search. The for loop is hardcoded to perform exactly 100 iterations. When designing $S$ you proved that it is impossible for the program search to find a malicious program that could take over the whole system. However, this proof makes the assumption that the for loop only performs exactly 100 iterations.

Assume that when performing more iterations in the program, it gives you better results. Then it seems plausible that an AGI would self-modify by increasing the number of iterations, increasing the risk of takeover.[^sr4farowwwh]

 If you don't have the optimal AGI source code, you need to put in active effort in order to ensure that the AGI will not self-modify, to ensure that your reasoning about the system doesn't break. Note that then the "don't perform self-modification" constraint is another part of your system that can break under optimization pressure.

My model about why people think that decision theory is important says: If we could endow the AGI with the optimal decision theory it would not self-modify, making it easier to reason about it.

If your AGI uses a bad decision theory $T$ it would immediately self-modify to use a better one. Then all your reasoning that uses features from $T$ goes out the window.

I also expect that decision theory would help you reason about how to robustly make a system non-self-modifying, but I am unsure about that one.

I am probably missing other reasons why people care about decision theory.

[^42imbsrhqz4]: This seems roughly right, but it's a bit more complicated than this. For example, consider the environment where it is optimal to change your source code according to a particular pattern, as otherwise you get punished by some all-powerful agent. 

[^sr4farowwwh]: One can imagine an AI that does not care whether it will be taken over, or one that can't realize that it would be taken over.
# A bet for Samo Burja

I'm listening to Samo Burja talk on the Cognitive Revolution podcast with Nathan Labenz.
Samo said that he would bet that AGI is coming perhaps in the next 20-50 years, but not in the next 5.

I will take that bet. I can't afford to make an impressively large bet because my counterfactual income is already tied up in a bet against the universe. I quit my well-paying industry job as a machine learning engineer / data scientist three years ago to focus on AI safety/alignment research. To make the bet interesting, I will therefore offer 10:1 odds.
I bet $1000 USD against your $100 USD that AGI will be invented in the next 5 years. There are a lot of possible resolution criteria, but as a reasonable shelling point I'll accept this metaculus market: https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/  

I'll describe my rationale here, in case I change your mind and make you not want the bet. ;-)

I agree with your premise that AGI will require fundamental scientific advances beyond currently deployed tech like transformer LLMs.

 I agree that scientific progress is hard, usually slow and erratic, fundamentally different from engineering or bringing a product to market.

I agree with your estimate that the current hype around chat LLMs, and focus on bringing better versions to market, is slowing fundamental scientific progress by distracting top AI scientists from pursuit of theoretical advances.

My cruxes are these:

1. I believe LLMs will scale to close enough to AGI to become central parts of very useful tools. I believe that these tools will enable the human AI scientists to make rapid theoretical progress. I expect that these AI research systems (I won't say researchers, since in this scenario they are still sub-AGI) will enable massively parallel testing of hypotheses which are derived as permutations of a handful of initial ideas given by the human scientists. I also foresee these AI research systems mining existing scientific literature for hypotheses to test. I believe the result of this technology will be rapid discovery of algorithms that can actually scale to true AGI.

2. I have been following advances in neuroscience relevant to brain-inspired AI for over 20 years now. I believe that the neuroscience community has made some key breakthroughs in the past five years which have yet to be effectively exported to machine learning and tested at scale. I also believe there's a backlog of older neuroscience findings that also haven't been fully tested. Thus, I believe the existing neuroscience literature provides a rich source of testable under-explored hypotheses. This could be tackled rapidly by the AI research systems from point 1, or will eventually be digested by eager young scientists looking for an academic ML paper to kickstart their careers. Thus the two cruxes are independent but potentially highly synergistic.

I look forward to your response!
Regards, Nathan Helm-Burger
# Is this voting system strategy proof?

My voting system works like this. Each voter expresses their preferences for all candidates on a real numbered utility scale. 

Then a Maximal lottery takes place over all lotteries over candidates. [https://en.wikipedia.org/wiki/Maximal_lotteries](https://en.wikipedia.org/wiki/Maximal_lotteries)

Lets describe this in more detail. Suppose there are 3 candidates. A,B,C. 

The set of candidates is $S=\{A,B,C\}$

A probability distribution over candidates looks like $(A:30\%,B:20\%,C:50\%)$

This probability distribution is in $\Delta S$ the set of all probability distributions over $S$.

A probability distribution over probability distributions looks like $((A:30\%,B:20\%,C:50\%) :60\%,(A:70\%,B:30\%,C:0\%):40\%)$

Though note that there are infinitely many distributions, so most distributions-of-distributions will be assigning probability densities. 

Also note that we can sample a candidate from this distribution over distributions by first sampling a distribution, and then sampling a candidate from that distribution. This is equivalent to integrating a distribution-of-distributions into a distribution over candidates and then sampling that. 

A distribution is equivalent to a point in a triangle.  A distribution over distributions is a probability density over a triangle, ie a non-negative function over the triangle (may include dirac deltas) 

So the voters all mark their preferences on a numerical scale. 

Then these votes get sent to Fred and George, 2 perfectly rational players in a 0 sum game. 

Fred and George both propose probability distributions over the candidates. 

Fred's utility is the number of candidates that strictly prefer Fred's proposed probability distribution over Georges, minus the number of voters that strictly prefer Georges distribution over Freds. 

This game has a unique Nash equilibrium. This equilibrium is a distribution over distributions. Sample a candidate from this equilibrium to get the election winner. 

I know that this has a few nice properties. If candidate $A$ is the first choice of the majority, then $A$ definitely wins. If everyone prefers A to B, then B has no chance of winning. If C has no chance of winning, the candidates existence doesn't influence the election. 

Is this system strategy proof, or can it be gamed? Will voters ever be incentivized to lie about their preferences?
# Pay Risk Evaluators in Cash, Not Equity

Personally, I suspect the alignment problem is hard. But even if it turns out to be easy, survival may still require getting at least the absolute basics right; currently, I think we're mostly failing even at that.

Early discussion of AI risk often focused on debating the viability of various elaborate safety schemes humanity might someday devise—designing AI systems to be more like “tools” than “agents,” for example, or as purely question-answering oracles locked within some kryptonite-style box. These debates feel a bit quaint now, as AI companies race to release agentic models they barely understand directly onto the internet.

But a far more basic failure, from my perspective, is that at present nearly all AI company staff—including those tasked with deciding whether new models are safe to build and release—are paid substantially in equity, the value of which seems likely to decline if their employers stop building and releasing new models.

As a result, it is currently the case that roughly everyone within these companies charged with sounding the alarm risks personally losing huge sums of money if they do. This extreme conflict of interest could be avoided simply by compensating risk evaluators in cash instead.
# I want a good multi-LLM API-powered chatbot

Chatbots are pretty useful to me, but the pro plans are expensive. I don't want to spend 20$/month here and there. However, I couldn't find a free chatbot app with at least OpenAI and Anthropic API support with a similar level of functionality as the online gpt or claude chatbots. In particular I want to attach images and pdfs.

The best thing I found so far is chatbox, but attaching documents is supported only with... a 20$/plan, and it seems the docs are sent to their server.

Am I spoiled to expect a free open-source software for anything? Though that's often the case for much more complicated applications I regularly use.

Does anyone know of something better?
# Lighthaven Sequences Reading Group #2 (Tuesday 09/17)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll post ahead of time with the food so you know if it works for you, but it will cover meat-eaters, veggies and vegans alike. Please RSVP to this event so we know how many people to have food for.

We're working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [The Meditation on Curiosity](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/3nZMgRTfFEfHp34Gb)
2.  [The Importance of Saying "Oops"](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/wCqfCLs8z5Qw4GbKS)
3.  [Tsuyoku Naritai](https://www.lesswrong.com/posts/DoLQN5ryZ9XkZjq5h/tsuyoku-naritai-i-want-to-become-stronger)
4.  [The Martial Art of Rationality](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/teaxCFgtmCQ3E9fy8)
5.  Optional: [Humans are not automatically strategic](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/PBRWb2Em5SNeWYwwB)

These are quite short and should take around 15-20 minutes to read all together.

The meetup starts at 6pm. We'll split into discussion groups around 6:30, and dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is going to be a regular meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Has Anyone Here Consciously Changed Their Passions?

I'm sorry if this question is a bit fuzzy, maybe my understanding of a "passion" isn't the most coherent. But, I'd like to know if anyone has made a successful effort to get themselves to be passionate about doing X where they were not passionate about X before. I suppose you could call that "creating drive."

For context: I have spent almost 25% of my life, maybe more, starting long term projects without ever finishing them. Often I find that whatever idea I've had or course of action I've plotted is conceptually cool, the outcomes look desirable, and hell, sometimes I even make decent progress -- but I can never seem to stick it through. There could be any number of reasons for this, but I think that a large part of my plight boils down to the fact that I am lacking in passion, which would presumably drive me to overcome small obstacles and keep me from falling off the course.

So, I am led to ask; has anyone here seen success in *deciding* to be passionate about something, or making the *decision* to enjoy working towards a particular goal?
# Building an Inexpensive, Aesthetic, Private Forum

**Reason for Writing**: I want to gather information for a friend who is a professional in the behavioral sciences.  
**Epistemic Status:** Less than 2 hours googling how to build a forum, and a couple youtube videos

**Requirements:**

1.  **Aesthetic**
2.  **Inexpensive**
3.  **Private**
4.  **Easily Setup**

The goal is to create a small, private forum for discussions. 

I've looked into discourse, not sure if that's the best option.
# Simon DeDeo on Explore vs Exploit in Science

Good news everyone: I’m getting into podcasting! 

As part of my investigation into how scientific paradigms are created, I hosted a talk by [Simon DeDeo, Professor Of Many Topics](https://sites.santafe.edu/~simon/) at Carnegie Mellon and the Santa Fe Institute, recorded at the recent ILIAD conference in Berkeley. He discusses the idea of explore vs exploit in science. Spoiler: he comes down very hard on the exploit side, based primarily on a case study of Darwin. 

For the video averse among you, here’s the [transcript](https://docs.google.com/document/d/1tdpwRrSbY02npnIGivKLKWVQrseBWhN-/edit). 

I have two more episodes in editing, but am on the hunt for more guests. I’m looking to talk to scientists or ex-scientists about how ideas are created and spread in their field. Bonus points if they can talk about Chaos, Complexity, or Catastrophe Theory in particular. 

Thanks to my Lightspeed Grants and my [Patreon patrons](https://www.patreon.com/acesounderglass) for supporting this work.
# Formalizing the Informal (event invite)

**Formalizing the Informal**
============================

One way to view MIRI's Agent Foundations research is that it saw the biggest problem in AI safety as "human preferences are informal, but we need to somehow get formal guarantees about them" -- and so, in response, it set out to make a formal-informal bridge.

Recently, I’ve been thinking about how we might formally represent the difference between formal and informal. My prompt is something like: if we assume that classical probability theory applies to “fully formal” propositions, how can we generalize it to handle “informal” stuff?  
  
I’m going to lead a discussion on this tomorrow, Wednesday Sept. 11, at 11am EDT (8am Pacific, 4pm UK).

Discord Event link (might not work for most people):

[https://discord.com/events/1237103274591649933/1282859362125352960](https://discord.com/events/1237103274591649933/1282859362125352960)

Zoom link (should work for everyone):

[https://us06web.zoom.us/j/6274543940?pwd=TGZpY3NSTUVYNHZySUdCQUQ5ZmxQQT09](https://us06web.zoom.us/j/6274543940?pwd=TGZpY3NSTUVYNHZySUdCQUQ5ZmxQQT09)

* * *

***You can*** [***support my work on Patreon***](https://www.patreon.com/AbramDemski/membership)***.***
# When can I be numerate?

I recently started working in a setting making physical products, where I discovered that I had a whole nother section of my brain purposed for physical interactions between things and when things might fall on other things or melt or fall apart or shatter or scar or any number of other things. Unfortunately, because of my inexperience, this is a part of my brain I had to be prompted to use. I had to be asked directly: "What do you think will happen as a result of that?" in order to actually use that part of my brain.

I can't help but feel that the same thing must be going on with my math ability.

I want to apply my math knowledge to the world so I can actually get some use out of it, but I have no idea how to go about doing that.

Wat do?
# OpenAI o1

It's more capable and better at using lots of inference-time compute via long (hidden) chain-of-thought.

**OpenAI pages:  **[Learning to Reason with LLMs](https://openai.com/index/learning-to-reason-with-llms/), [o1 System Card](https://openai.com/index/openai-o1-system-card/), [o1 Hub](https://openai.com/o1/)

**Tweets:** [Sam Altman](https://x.com/sama/status/1834283100639297910), [Noam Brown](https://x.com/polynoamial/status/1834280155730043108), [OpenAI](https://x.com/OpenAI/status/1834278223775187374)

**Discussion:  **[https://www.transformernews.ai/p/openai-o1-alignment-faking](https://www.transformernews.ai/p/openai-o1-alignment-faking)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/830ca00b9969f92d2487701feeebf4dadc67d3f8bcbcecc2.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/33239d50232c061ef4edf123a36e233fb0f8a258da9b887c.png)

"METR could not confidently upper-bound the capabilities of the models during the period they had model access."

"Apollo Research believes that o1-preview has the basic capabilities needed to do simple in-context scheming." (Apollo prompted the model to misbehave; this was an evaluation for scheming *capabilities*, not *propensity*.)

> Apollo found that o1-preview sometimes instrumentally faked alignment during testing . . . , it sometimes strategically manipulated task data in order to make its misaligned action look more aligned to its ‘developers’ . . . , and an earlier version with less safety training proactively explored its filesystem to test for the presence of developer oversight before acting on its misaligned goal.
# Why I'm bearish on mechanistic interpretability: the shards are not in the network

Once upon a time, the sun let out a powerful beam of light which shattered the world. The air and the liquid was split, turning into body and breath. Body and breath became fire, trees and animals. In the presence of the lightray, any attempt to reunite simply created more shards, of mushrooms, carnivores, herbivores and humans. The hunter, the pastoralist, the farmer and the bandit. The king, the blacksmith, the merchant, the butcher. Money, lords, bureaucrats, knights, and scholars. As the sun cleaved through the world, history progressed, creating endless forms most beautiful.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/16479f9726f28346101dba1c6038ef8b00623c5385151cee.webp)

It would be perverse to try to understand a king in terms of his molecular configuration, rather than in the contact between the farmer and the bandit. The molecules of the king are highly [diminished](https://www.lesswrong.com/s/gEvTvhr8hNRrdHC62/p/7YwxaX7qzQfBDmxzR) phenomena, and if they have information about his place in the ecology, that information is widely spread out across all the molecules and easily lost just by missing a small fraction of them. Any thing can only be understood in terms of the greater forms that were shattered from the world, and this includes neural networks too.

But through gradient descent, shards act upon the neural networks by leaving imprints of themselves, and these imprints have no reason to be concentrated in any one spot of the network (whether activation-space or weight-space). So studying weights and activations is pretty doomed. In principle it's more relevant to study how external objects like the dataset influence the network, though this is complicated by the fact that the datasets themselves are a mishmash of all sorts of random trash[^p7hz04evn4o].

Probably the most relevant approach for current LLMs is [Janus](https://x.com/repligate)'s, which focuses on how the different styles of "alignment" performed by the companies affect the AIs, qualitatively speaking. Alternatively, when one has scaffolding that couples important real-world shards to the interchangeable LLMs, one can study how the different LLMs channel the shards in different ways.

Admittedly, it's very plausible that future AIs will use some architectures that bias the representations to be more concentrated in their dimensions, both to improve interpretability and to improve agency. And maybe mechanistic interpretability will work better for such AIs. But we're not there yet.

[^p7hz04evn4o]: Possibly clustering the data points by their network gradients would be a way to put some order into this mess? But two problems: 1) The data points themselves are merely diminished fragments of the bigger picture, so the clustering will not be properly faithful to the shard structure, 2) The gradients are as big as the network's weights, so this clustering would be epically expensive to compute.
# OpenAI o1, Llama 4, and AlphaZero of LLMs

GPT-4 level open weights models like Llama-3-405B don't seem capable of dangerous cognition. OpenAI o1 demonstrates that a GPT-4 level model can be post-trained into producing useful long horizon reasoning traces. AlphaZero shows how capabilities can be obtained from compute alone, with no additional data. If there is a way of bringing these together, the apparent helplessness of the current generation of open weights models might prove misleading.

Post-training is currently a combination of techniques that use synthetic data and human labeled data. Human labeled data significantly improves quality, but its collection is slow and scales poorly. Synthetic data is an increasingly useful aspect of post-training, and automated aspects of its generation scale easily. Unlike weaker models, GPT-4 level LLMs clearly pass reading comprehension on most occasions, OpenAI o1 improves on this further. This suggests that at some point [human data might become mostly unnecessary in post-training](https://www.interconnects.ai/p/ai-data-foundry), even if it still slightly helps. Without it, post-training becomes automated and gets to use more compute, while avoiding the need for costly and complicated human labeling.

A pretrained model at the next level of scale, such as Llama 4, if made available in open weights, might initially look approximately as tame as current models. OpenAI o1 demonstrates that useful post-training for long sequences of System 2 reasoning is possible. In the case of o1 in particular, this might involve a lot of human labeling, making its reproduction a very complicated process (at least if the relevant datasets are not released, and the reasoning traces themselves are not leaked in large quantities). But if some generally available chatbots at the next level of scale are good enough at automating labeling, this complication could be sidestepped, with o1 style post-training cheaply reproduced on top of a previously released open weights model.

So there is an overhang in an open weights model that's distributed without long horizon reasoning post-training, since applying such post-training significantly improves its capabilities, making perception of its prior capabilities inadequate. The problem right now is that a new level of pretraining scale is approaching in the coming months, while ability to cheaply apply long horizon reasoning post-training might follow shortly thereafter, possibly unlocked by these very same models at the new level of pretraining scale (since it might currently be too expensive for most actors to implement, or to do enough experiments to figure out how). The resulting level of capabilities is currently unknown, and could well remain unknown outside the leading labs until *after* the enabling artifacts of the open weights pretrained models at the next level of scale have already been published.
# AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space

Authors: [Huzheng Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+H), [James Gee](https://arxiv.org/search/cs?searchtype=author&query=Gee,+J), [Jianbo Shi](https://arxiv.org/search/cs?searchtype=author&query=Shi,+J).

Abstract:

> We study the intriguing connection between visual data, deep networks, and the brain. Our method creates a universal channel alignment by using brain voxel fMRI response prediction as the training objective. We discover that deep networks, trained with different objectives, share common feature channels across various models. These channels can be clustered into recurring sets, corresponding to distinct brain regions, indicating the formation of visual concepts. Tracing the clusters of channel responses onto the images, we see semantically meaningful object segments emerge, even without any supervised decoder. Furthermore, the universal feature alignment and the clustering of channels produce a picture and quantification of how visual information is processed through the different network layers, which produces precise comparisons between the networks.
# Superintelligence Can't Solve the Problem of Deciding What You'll Do

Superintelligence can solve all problems, even as it won't necessarily do so. But not quite. In the modern world, you [exist within physics](https://www.lesswrong.com/posts/NEeW7eSXThPz7o4Ne/thou-art-physics), and all you do obeys its laws. Yet it's still you who decides what you do. If there is a simulation of your behavior, that doesn't change the attribution of the reason for it. When the fabric of reality is woven from superintelligent will rather than only physical law, the reason for your own decisions is still you, and it's still not possible for something that is not you to make decisions that are yours.

Potential for manipulation or physical destruction doesn't distinguish the role of superintelligence from that of the physical world. To make decisions in the world, you first need to exist, in an [environment](https://www.lesswrong.com/posts/8oMF8Lv5jiGaQSFvo/boundaries-part-1-a-key-missing-concept-from-utility-theory) where you are able to function. Being overwritten or changed into something else, either with brute force or subtle manipulation or social influence, is a form of not letting this premise obtain.

A [solved world](https://www.amazon.com/Deep-Utopia-Meaning-Solved-World/dp/1646871642) is post-instrumental, all things that are done for a purpose could be done by AI to more effectively reach that purpose. It could even be more effective than you at figuring out what your decisions are going to be! This is similar to what an optimizing compiler does, the behavior of machine code is still determined by the meaning of the source code, even if that source code is erased and only exists conceptually. With humans, the physical implementation is similarly not straightforward, all the proteins and synaptic vesicles are more akin to machine code than a conceptually reasonable rendering of a person. So it's fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect [simulators](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators).

In this framing, superintelligence is capable of being a better simulator of you, but it gets no headway in being capable of deciding your behavior. A puzzle associated with determinism is to point out that superintelligence can show you a verifiable formal proof of what you're going to do. But that doesn't really work, since deciding to ignore the proof and doing whatever makes the situation you observed go away as counterfactual, while in actuality you are still determining your own actions. Only if you do have the property of deferring to external proofs of what you'll do, will it become the case that an external proof claiming something about your behavior is the reason for that behavior. But that's [no way to live](https://www.lesswrong.com/posts/YrhT7YxkRJoRnr7qD/no-license-to-be-human). Instead, it's you who controls the course of proofs about your behavior, not those who are writing the proofs.
# If I wanted to spend WAY more on AI, what would I spend it on?

Supposedly intelligence is some kind of superpower.  And they're now selling intelligence for pennies/million tokens.  Logically, it seems like I should be spending [way more](https://x.com/kellerjordan0/status/1834990147555017180) of my income than I currently am on intelligence.  But what should I spend it on?

For context, I currently spend ~$50/month on AI:

*   ChatGPT $20/month
*   Github Copilot $10/month
*   Various AI art apps ~$20/month

Suppose I wanted to spend *much* more on intelligence (~$1000/month), what should I spend it on?

One idea might be: buy a pair of [smart glasses](https://www.ray-ban.com/usa/ray-ban-meta-smart-glasses), record everything I do, dump it into a database, and then have the [smartest LLM I can find](https://openai.com/index/introducing-openai-o1-preview/) constantly suggest things to me based on what it sees.

Is this the best thing I could do?

Would it be anywhere worth $1000/month (assume spending this money will not impact my welfare in any way and I would otherwise dump it into an SNP500 index fund).
# Hyperpolation

Interpolation, Extrapolation, Hyperpolation: Generalising into new dimensions

by [Toby Ord](https://en.wikipedia.org/wiki/Toby_Ord) 

Abstract:

> This paper introduces the concept of hyperpolation: a way of generalising from a limited set of data points that is a peer to the more familiar concepts of interpolation and extrapolation. **Hyperpolation is the task of estimating the value of a function at new locations that lie outside the subspace (or manifold) of the existing data.** We shall see that hyperpolation is possible and explore its links to creativity in the arts and sciences. We will also examine the role of hyperpolation in machine learning and suggest that the lack of fundamental creativity in current AI systems is deeply connected to their limited ability to hyperpolate.

And it turns out that it is as doable systematically as extrapolation. For example, before reading the paper, can you guess an f(x,y) that is a simple reasonable generalization of this f(x)?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9af59e33d9f23b1721c1dfaa9cc35fee43bfdbe167ce9fc1.png)
# Can subjunctive dependence emerge from a simplicity prior?

Suppose that an embedded agent models its environment using an approximate simplicity prior, would it acquire a [physicalist agent ontology or an algorithmic/logical agent ontology](https://www.lesswrong.com/posts/dmjvJwCjXWE2jFbRN/fdt-is-not-directly-comparable-to-cdt-and-edt#The_2x2x2_decision_theory_grid)?

One argument for the logical agent ontology is that it allows the agent to compress different parts of its observations that are subjunctively dependent: If two physical systems are computing the same function, the logical agent ontology only has to store that function once, then model the dependencies between those two systems and the function. On the other hand, information about that function would be redundantly represented in both physical systems under the physicalist agent ontology.

Most decision theory literature seems to treat (physical) causal dependence as the default, requiring extra work to formalize subjunctive dependence. However, if logical agent ontology naturally emerges from a simplicity prior, we might expect subjunctive dependence to arise by default for most agents.

*Thanks to Anthony Digiovanni for the discuission that inspired this post*
# Does life actually locally *increase* entropy?

The usual materialist story of life I've heard is that life acts like an entropy pump, creating local reductions of entropy within the organism but increasing the entropy outside of the organism. (I think I've even seen that in The Sequences somewhere? But couldn't find it, feel encouraged to link it.) But I've come to think that might actually be wrong and life might increase entropy both inside and outside the organism.

Here's a rough account:

*   We ought to expect entropy to increase, so *a priori* life is much more feasible if it increases entropy rather than decreasing entropy.
*   Living matter is built mainly out of carbon and hydrogen, which is extracted from CO2 and H2O, leaving O2 as a result. Entropy breakdown:
    *   The O2 left over from breaking up CO2 ought to have somewhat *lower* entropy than the original CO2.
    *   The O2 left over from breaking up the original H2O ought to have... higher entropy because it's a gas now?
    *   The hydrocarbons don't have much entropy because they stick together into big chunks that therefore heavily constrain their DOFs, but they do have *some* entropy for various reasons, and they are much more tightly packed than air, so per volume they oughta have orders of magnitude more entropy density. (Claude estimates around 200x.)
    *   Organic matter also traps a lot of water which has a high entropy density.
    *   Usually you don't talk about entropy *density* rather than absolute entropy, but it's unclear to me what it means for organisms to "locally" increase/decrease entropy if not by density.
*   Oxygen + hydrocarbons = lots of free energy, while water + carbon dioxide = not so much free energy. We usually associate free energy with low entropy, but that's relative to the burned state where the free energy has been released into thermal energy. In this case, we should instead think relative to an unlit state where the energy hasn't been collected at all. Less energy generally correlates to lower entropy.

Am I missing something?
# Where to find reliable reviews of AI products?

Being able to quickly incorporate AI tools seems important, including for working on AI risk (people who disagree: there's a thread for doing so in the comments). But there are a lot of AI products and most of them suck.  Does anyone know a good source of reviews, or even just listing product features and naming obvious slop?
# Which LessWrong/Alignment topics would you like to be tutored in? [Poll]

Would you like to be tutored in [applied game theory](https://www.lesswrong.com/tag/game-theory), [natural latents](https://www.lesswrong.com/posts/mMEbfooQzMwJERAJJ/natural-latents-the-concepts), [CFAR-style rationality techniques](https://www.lesswrong.com/posts/dbDHEQyKqnMDDqq2G/cfar-handbook-introduction), "general AI x-risk", [Agent Foundations](https://www.lesswrong.com/tag/agent-foundations), [anthropic**s**](https://www.lesswrong.com/tag/anthropics), or some [other topics](https://www.lesswrong.com/concepts) discussed on LessWrong?

I'm thinking about prototyping some topic-specific LLM tutor bots, and would like to prioritize topics that multiple people are interested in.

Topic-specific LLM tutors would be customized with things like pre-loaded relevant context, helpful system prompts, and more focused testing to ensure they work.

**Note: I'm interested in topics that are written about on LessWrong, e.g. infra-bayesianism, and** ***not*** **magnetohydrodynamics".**

* * *

I'm going to use the same poll infrastructure that [Ben Pace pioneered](https://www.lesswrong.com/posts/WEbSrDuLhqrtP5uuH/vote-on-anthropic-topics-to-discuss) recently. There is a [**thread below**](https://www.lesswrong.com/posts/Hb9LGGyv4RqWE9jBq/which-lesswrong-alignment-topics-would-you-like-to-be?commentId=WNQ9kLhD4zckEm9Gi) where you add and vote on topics/domains/areas where you might like tutoring.

1.  **Karma:** upvote/downvote to express enthusiasm about there being tutoring for a topic.
2.  **Reacts:** click on the **agree react** to indicate you personally would like tutoring on a topic.
3.  **New Poll Option.** Add a new topic for people express interest in being tutored on.

For the sake of this poll, I'm more interested in whether you'd like tutoring on a topic or not, separate from the question of whether you think a tutoring bot would be any good. I'll worry about that part.

**Background**

I've been playing around with LLMs a lot in the past couple of months and so far my favorite use case is tutoring. LLM-assistance is helpful via multiple routes such as providing background context with less effort than external search/reading, keeping me engaged via interactivity, generating examples, and breaking down complex sections into more digestible pieces.
# To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

Authors: [Zayne Sprague](https://arxiv.org/search/cs?searchtype=author&query=Sprague,+Z), [Fangcong Yin](https://arxiv.org/search/cs?searchtype=author&query=Yin,+F), [Juan Diego Rodriguez](https://arxiv.org/search/cs?searchtype=author&query=Rodriguez,+J+D), [Dongwei Jiang](https://arxiv.org/search/cs?searchtype=author&query=Jiang,+D), [Manya Wadhwa](https://arxiv.org/search/cs?searchtype=author&query=Wadhwa,+M), [Prasann Singhal](https://arxiv.org/search/cs?searchtype=author&query=Singhal,+P), [Xinyu Zhao](https://arxiv.org/search/cs?searchtype=author&query=Zhao,+X), [Xi Ye](https://arxiv.org/search/cs?searchtype=author&query=Ye,+X), [Kyle Mahowald](https://arxiv.org/search/cs?searchtype=author&query=Mahowald,+K), [Greg Durrett](https://arxiv.org/search/cs?searchtype=author&query=Durrett,+G).

Abstract:

> Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

X(/twitter) thread: [https://x.com/ZayneSprague/status/1836784332704215519](https://x.com/ZayneSprague/status/1836784332704215519).
# Lighthaven Sequences Reading Group #3 (Tuesday 09/24)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [The Bottom Line](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line)
2.  [Rationalization](https://www.lesswrong.com/posts/SFZoEBpLo9frSJGkc/rationalization)
3.  [You Can Face Reality](https://www.lesswrong.com/posts/HYWhKXRsMAyvRKRYz/you-can-face-reality)
4.  [Is That Your True Rejection?](https://www.lesswrong.com/posts/TGux5Fhcd7GmTfNGC/is-that-your-true-rejection)
5.  [Avoiding Your Belief's Real Weak Points](https://www.lesswrong.com/posts/dHQkDNMhj692ayx78/avoiding-your-belief-s-real-weak-points)

These are quite short and should take around 15-20 minutes to read all together.

The meetup starts at 6pm. We'll split into discussion groups around 6:30, and dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is going to be a regular meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# I'm creating a deep dive podcast episode about the original Leverage Research - would you like to take part?

Hi all! I'm creating a deep dive podcast episode about (the original) Leverage Research (for my podcast, [Clearer Thinking](https://podcast.clearerthinking.org/)). If you worked at (or currently work at) Leverage, or you have had experiences connected to Leverage, and you would like to either potentially:

(1) do an interview for the podcast

(2) talk to me totally off the record about your experiences (i.e., with your privacy totally protected)

Please reach out privately by using [my contact form here](https://www.spencergreenberg.com/contact-spencer/). Or you can email our podcast (but multiple team members who work on the podcast will see it) here:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c549a9dfe165375b4411dc64bfaeea28ab82803b4dba4370.png)

I have known many people involved in Leverage over the years (from its founding to the present day), and so I have heard a great deal of information about it from others. But I've never been a member of it, and there are still many things I don't know about it. 

The podcast episode aims to provide a deep exploration (that's as unbiased as we can achieve) into (i) what Leverage was, (ii) what sorts of activities, occurrences, findings, and developments happened there, and (iii) why it ended up falling apart.

Note that Leverage still exists in a modified form, but the focus of the podcast will be on "Leverage 1.0."

Note also that while there are some interviews that I will be able to conduct right away, I will have to wait a while before conducting others (if yours falls into that camp, I will let you know).

Of course, I won't necessarily end up interviewing everyone who reaches out, but I'd be interested to hear from you regardless.