Harley-ml/lesswrong huggingface dataset:
# Shh, don't tell the AI it's likely to be evil

If you haven't been online in the past week: [ChatGPT](https://openai.com/blog/chatgpt/) is OpenAI's new fine-tuned language model that is really much better at doing all sorts things that you'd expect. Here's it [debugging some code](https://twitter.com/amasad/status/1598042665375105024), or [writing awful-sounding Seinfeld episodes](https://twitter.com/goodside/status/1598077257498923010), or (my favorite) writing a verse from [the King James Bible about taking a PB sandwich out of a VCR.](https://twitter.com/tqbf/status/1598513757805858820)

My fairly-basic understanding of large language models is that they are trying to predict the most-likely-next token in a sequence. So if you start a sentence with "I love peanut butter and..." the large language model will finish it with "jelly." It's clear that the relationships be captured by "most-likely-token" are way more complex and interesting that one might initially guess (see: getting a PB sandwich out of a VCR above).

Given that LLMs are trained on the public internet, I'm wondering if we should be thinking hard about what sorts of sequences we are exposing these models to.

Imagine OpenAI of the future goes to fine-tune one of their models to be a "friendly and nice and happy chatbot that just wants to help humans solve their problems, while also being aligned + not evil." ChatGPT5 is born, and when given a question as a prompt, it will likely respond with something friend and nice and happy, etc. 

But deep in the weights in ChatGPT5's network are a memory of the other "friendly and nice and happy chatbots" it has seen before: the AIs that we have been writing about since Asimov. And most of the AIs in our fiction have a tendency to work great at first, but eventually turn evil, and then try to kill everyone.

And so, as these are the only other chatbots that ChatGPT5 has learned from, this may become the most-likely sequence. Be a nice and friendly and happy chatbot - and then, after about 2 years of being friendly and visibly-aligned, go ham -- for no other reason than this is the most likely set of tokens it's seen.

Another way to ask this question that: would you rather ChatGPT had been trained on the dataset it was likely trained on (e.g. human vs. AI war), or one where every story about an AI was turned to one of solar punk abundance and happy utopia?

To me, the answer is pretty clear. 

The solution, less so. I wonder if I should devote more time to writing aligned-AI fiction and putting on the internet where it can be found by OpenAI's web scraper. Not by a human though -- all the fiction I write is god awful.
# Verification Is Not Easier Than Generation In General

People who’ve spent a lot of time thinking about [P vs NP](https://en.wikipedia.org/wiki/P_versus_NP_problem) often have the intuition that “verification is easier than generation”. It’s easier to verify a solution to some equations than to find a solution. It’s easier to verify a password than to guess it. That sort of thing. The claim that it is easier to verify solutions to such problems than to generate them is essentially the claim that P ≠ NP, a conjecture which is widely believed to be true. Thus the intuition that verification is generally easier than generation.

The problem is, this intuition comes from *thinking about problems which are in NP*. NP is, roughly speaking, the class of algorithmic problems for which solutions are easy to verify. Verifying the solution to some equations is easy, so that problem is in NP.

I think a more accurate takeaway would be that *among problems in NP*, verification is easier than generation. In other words, among problems for which verification is easy, verification is easier than generation. Rather a less impressive claim, when you put it like that.

With that in mind, here is an algorithmic problem for which generation is easier than verification.

> Predicate: given a program, does it halt?
> 
> Generation problem: generate a program which halts.
> 
> Verification problem: given a program, verify that it halts.

The generation problem is trivial. The verification problem is uncomputable.

That’s it for the post, you all can argue about the application to alignment in the comment section.
# Things roll downhill

(Response to [this challenge](https://www.lesswrong.com/posts/tD9zEiHfkvakpnNam/a-challenge-for-agi-organizations-and-a-challenge-for-1))

I've read two things recently on similar strategies: [Zvi's post on ChatGPT](https://www.lesswrong.com/posts/RYcoJdvmoBbi5Nax7/jailbreaking-chatgpt-on-release-day), and [Scott Alexander's post on Redwood Research.](https://astralcodexten.substack.com/p/can-this-ai-save-teenage-spy-alex)

They both seem to have a similar strategy, to train AI to not do misaligned things by giving it a bunch of examples of misaligned things, and saying don't do that.

They both fail to out of training distribution errors. ChatGPT with asking it to pretend, or even just to speak in uwu voice, and Redwood with poetic descriptions, plus weirder things.

The Redwood post shows that they thought about this problem. They have a step for "find out of distribution errors, and train more, especially on them." Plus an idea that this could still have promise with more of these steps.

If they (optimistically) remove 90% of all edge cases every iteration, then after only infinite steps, they'll have removed them all.

You don't want to be playing Sisyphus against the AI.

The model wants to complete language, aligned or not. You're trying to push it away from things you don't like. But unless you push it in exactly the right direction, you're only vaguely pushing it uphill. There is exactly one stable point uphill, and it's the exact peak. If you aren't aiming *perfectly* you're going to miss it.

And the ball will roll down the hill.

(Plus there's a good chance alignment isn't at the peak, and is just a random unstable point on the side of the hill)

The ball will always roll down the hill. The goal is not to push the ball. It's to redesign the hill. Turn it into a valley.

Ideas like [Mechanistic Anomaly Detection](https://www.lesswrong.com/posts/vwt3wKXWaCvqZyF74/mechanistic-anomaly-detection-and-elk) sound more promising. Naive implementations of something like that might fall to the same problems. If it's "train an AI to try to notice when we're out of distribution, then correct for it," we're just adding another Sisyphus who double checks we're pushing uphill.

If each extra Sisyphus catches 90% of mistakes, then...

The design needs to be a system that says "The AI can't go out of distribution, because that's *uphill*."
# MIRI's "Death with Dignity" in 60 seconds.

Suppose that, like Yudkowsky, you really care about humanity surviving this century but you think that nothing you can do has a decent chance of achieving that.

It's an unfortunate fact of human psychology that, when faced with this kind of situation, people will often do nothing at all instead of the thing which has the *highest chance* of achieving their goal. Hence, you might give up on alignment research entirely, and either lie in bed all day with paralysing depression, or convert your FAANG income into short-term pleasures. How can we avoid this trap?

It seems we have three options:

*   **(1) Change your psychology.** This would be the ideal option. If you can do that, then do that. But the historical track-record suggests this is really hard.
*   **(2) Change your beliefs.** This is called "hope", and it's a popular trick among AI doomers. You change your belief from "there's nothing I can do which makes survival likely" to "there's something I can do which makes survival likely".
*   **(3) Change your goals.** [This is what Yudkowsky proposes.](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) You change your goal from "humanity survives this century" to "my actions increase the log-odds that humanity survives this century". Yudkowsky calls this new goal "dignity". The old goal had only two possible values, $0$ and $1$, but the new goal has possible values anywhere between $-\infty$ and $+\infty$.

Of course, it's risky to change either your beliefs or your goals, because you might face a situation where the optimal policy after the change differs from the optimal policy before the change. But Yudkowsky thinks that (3) is less optimal-policy-corrupting than (2).

Why's that? Well, if you force yourself to believe something unlikely (e.g. "there's something I can do which makes survival likely"), then the inaccuracy can leak into your other beliefs because your beliefs are connected together by a web of inferences. You'll start making poor predictions about AI, and also make silly decisions.

On the other hand, changing your goal from "survival" to "dignity" is like [Trying to Try](https://www.lesswrong.com/posts/WLJwTJ7uGPA5Qphbp/trying-to-try) rather than trying — it's relatively less optimal-policy-corrupting.
# How do finite factored sets compare with phase space?

Garrabrant's [finite factored sets](https://www.lesswrong.com/tag/finite-factored-sets) feel to me like the same thing as a [phase space](https://en.wikipedia.org/wiki/Phase_space) of a dynamical system. The differences I can see are that phase spaces are not always finite, and that finite factored sets don't have the context of a dynamical rule defined on them. They seem to share the property that every element in the set has exactly one coordinate in each dimension/is an element of exactly one partition of each factor.
# Simple Way to Prevent Power-Seeking AI

The most concerning part of AGI is that it may have power-seeking tendencies. We can loosely define a powerful state to be a state that is useful to be in regardless of the reward function. A promising approach to AI safety is to make the RL process "interpretable" enough so that you can easily see which states the model regards as "powerful". 

The dumbest possible example of this is to train an RL agent on many tasks by learning a task-specific head for each task. Then, one can average the q-value across the different tasks -- if this number is high, the state is "powerful". Obviously, one can do much more sophisticated things. My point is that this is a fairly tractable research direction.

Then, when you sample from the agent during train/test time, avoid the powerful actions. 

Now, suppose the RL agent was fully sentient and knew you were doing the above. Does it have an incentive to deceive you -- e.g. make powerful actions seem not powerful? 

Well, suppose the RL agent was trained via Q-learning and it found some powerful actions that the above metric didn't recognize as powerful. Then, it would just update the Q value across different tasks. The RL agent knows that this update will allow us to recognize that this state is powerful, thus, we'll ensure this state isn't visited again. However, the RL agent is simply trying to accurately estimate the discounted expected reward given that we are in state s and are taking action a. The future probability of ending up at (s,a) isn't relevant to this -- thus, there's no incentive to be deceptive. 

But let's say the RL agent was trained via policy gradients. Now, it's trying to recommend the optimal action that maximizes the reward. Let's say this agent found a powerful action that the metric didn't recognize as powerful. The agent also knows that if it increases the probability of this action across different tasks, we are less likely to choose this action -- since we can recognize that it's a powerful action. The RL agent can maximize its reward by being deceptive and hiding that the action is powerful.
# ACX Montreal Meetup December 17 2022

**Venue: **

[Olde Orchard pub & grill](https://www.google.com/maps/place/Ye+Olde+Orchard+Pub+%26+Grill/@45.5134983,-73.5727146,17z/data=!3m1!4b1!4m5!3m4!1s0x4cc91a35812fae03:0xf5fdb66587011cc4!8m2!3d45.5135005!4d-73.5727198) at 20 Prince Arthur street W.  
Date & Time: December 17th, 2022 at 1pm.

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup. No readings this time, just a social meetup for the busy month of December to discuss recent blogs and curiosities.

Please try to RSVP as we will make a reservation at the restaurant.
# [ASoT] Probability Infects Concepts it Touches

*Epistemic status: Random ranting about things I'm confused about. Hopefully this post makes you more confused about optimization/agency/etc.*

A few parables of counterfactuals

*   What if $2+2$ equaled $5$?
    *   American: Hmm, Yeah. What if giving me two dollars and then another two had me end up with $5$? I can imagine that.
    *   Me: What! This can obviously never happen! This would break literally everything! It's literally a logical contradiction!
*   What if Quantum computers could solve NP-complete problems?
    *   Me: Hmm sounds cool, I can imagine-
    *   Scott Aaronson: What! They obviously can't because (*unintelligible*). You'd have to fundamentally alter how reality works, and even then things might not be salvageable.
*   What if there wasn't a storm today in Texas?
    *   Ancient Greek: I'm imagining Zeus wasn't as angry down, perhaps Hera calmed him down.
    *   Weather Expert: I knew there'd be a storm 3 days ago, if there *wasn't* a storm today, that would necessarily change the weather in the adjacent states over the past few days.
    *   Superintelligence: Uh... There had to be a storm today...

* * *

What is optimization?

Suppose you view optimization as "pushing the world into low-probability states." Consider the following

*   Does an asteroid perform optimization? We can predict its path years in advance, so how "low-probability" was the collision?
*   Many here would intuitively agree that Elon Musk is a powerful optimizer (pushed the world into low-probability states). Yet, a sufficiently powerful predictor wouldn't have been surprised by Tesla and SpaceX succeeding.
*   Does the bible perform optimization (insofar as the world looks different in the counterfactual without it)? Or does the "credit" go to its authors? (Same with an imaginary being everyone believes in)

Can we really say optimization is a thing in the territory? Or is it an artifact of the map?

The subjectiveness of probability "infects" all the concepts that use it. As do the limitations of the theory (like being compute-bounded; aka logical counterfactuals)

For example, if optimization is pushing the universe into subjectively unlikely states then all the confusion gets pushed into the word "pushing"[^2err3renab5].

* * *

This is what philosophy does to you. This is why bits of the universe shouldn't think too hard about themselves.

I suspect I've been nerdsniped by a wrong question somehow. This line of thought doesn't seem productive for aligning the AI... Curious to hear takes in the comments.

[^2err3renab5]: Pun intended.
# "Attention Passengers": not for Signs

On the MBTA, when a train is coming you get an announcement like:

> Attention passengers, the next red line train to Ashmont is now arriving.

As with all the announcements, there's a text version:

[![](https://www.jefftk.com/attention-passengers-the-next-red-line-train.jpg)](https://www.jefftk.com/attention-passengers-the-next-red-line-train-big.jpg)

[![](https://www.jefftk.com/to-ashmont-is-now-arriving.jpg)](https://www.jefftk.com/to-ashmont-is-now-arriving-big.jpg)

I like that they have the signs, both for general accessibility reasons and because you're often in a place where you can read the sign but not hear the announcement. But I don't like that they include "attention passengers".

Including those words in the audio version I understand: you need to catch peoples attention before you start giving them the information. On a sign, however, it's not adding anything. What makes it worse here is that the critical information, which direction the arriving train is traveling, is pushed onto the second screen. Someone who could have enough time to catch the train if they started hustling when the announcement first came up, might well not have enough time if they need to wait for the second screen about ten seconds later.

This isn't a problem for me anymore, now that I've set up my phone so that [checking train arrivals](https://www.jefftk.com/p/low-friction-mbta-predictions) is low-friction but it was minorly frustrating for years.

I suspect all it would take to fix this is editing some tiny file that stores the announcement text...

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02eQdQfZWcnXPKB9hRe5fc6RBozBEosNzd3sRgHJxCprF7HUc2A8wN2XTwyGnEQyobl), [mastodon](https://mastodon.mit.edu/@jefftk/109469904666826393)*
# Why I like Zulip instead of Slack or Discord

Slack and Discord are skins over the same thing. A bunch of conversations happening at the same time. Threads are made after the fact, and many conversations are braided together, so you have to untangle them.

Zulip has a tiny change: you have to make a conversation have a point up front by giving it a thread title. Random convos can happen in the 'random' thread, so it includes the previous model. 

This is so much nicer. Conversations are untangled and it becomes way easier to go through messages quickly. Conversations also **end**, instead of just petering out before oozing into another one. Threads introduce cutoff points.

Zulip mobile app is meh though.
# [Link] Wavefunctions: from Linear Algebra to Spinors

I wrote this blogpost because I thought it took an excessive amount of digging to understand what a spinor was. My original motivation was to understand wavefunctions more concretely since I recently discovered that wavefunctions are spinor-valued, not (necessarily) complex-valued. That took me down a rabbit hole of gamma matrices, geometric algebra, quaternions, and about a dozen other topics.

I think physics is taught very badly. Modern physical theories are built on some very heavy and very powerful mathematical machinery. That machinery is absolutely worth learning, but expositions on physical phenomena seem to have no middle ground between "breadth-first" (require all the background before being able to understand anything), "assembly-level" (discuss the raw equations without any intuition), and "vague analogies." It seems entirely possible to introduce slices of very abstract math as needed so people can go deep without having to go wide and without having to sacrifice either intuition or precision.

Anyway. This blogpost was a proof of concept. It assumes a background in linear algebra, no more than what's taught to a STEM freshman. I try to explain a vertical slice of the mathematical machinery needed to understand spinors. I'm not a physicist, nor do I have access to one, so I might have gotten something wrong. If you notice any errors, please let me know.
# Looking for ideas of public assets (stocks, funds, ETFs) that I can invest in to have a chance at profiting from the mass adoption and commercialization of AI technology

Playing around with ChatGPT made me realize that it is quite likely that the next profitable tech breakthrough will probably come from AI technology. What do I mean by this? every few years there is a breakthrough (or a few) in technology that gets massively adopted, gets monetized, and becomes very lucrative and profitable to its owner. In the early 2000s, it was e-commerce and online advertisement. In the late 2000s, it was the smartphone. In the early 2010s it was the subscription model service. In the late 2010s, it was SaaS.  
  
Basically I want to position part of my investment portfolio in a way that I have exposure to the next Apple, Amazon, or Salesforce, which I think will come from a successful monetization of AI technology.   
  
I already own LRNZ ([TrueShares AI & Deep Learning Fund](https://truesharesetfs.com/lrnz)) and SMH ([VanEck Semiconductor ETF](https://www.vaneck.com/us/en/investments/semiconductor-etf-smh/))  
  
I could also increase my exposure to GOOG, META, and MSFT, as they seem to be the public companies best positioned for an AI breakthrough that can be monetized. 

I know AI uses quite a bit of computer power so perhaps something like NVDA also makes sense?

Any other ideas?
# I've started publishing the novel I wrote to promote EA

Currently I'm just uploading it to Royal Road, but in the next week I'll also post it to the spacebattles forum, and around the start of January I'll publish a copy to Amazon. 

So, if you find this project interesting, and use Royal Road, follow it and maybe leave a comment there. Also, if there is a site you use to read and discover fiction that would be appropriate for me to publish this to, tell me, and I'd love to put my novel there.

If anyone can help in some way to tell people who they think might like a Isekai/portal fantasy novel about an effective altruist trying to do as much good as he can after he died and woke up with super magical powers in a fantasy world, please do so. 

Link: [https://www.royalroad.com/fiction/61682/the-split-summon/chapter/1058375/chapter-one-arrival](https://www.royalroad.com/fiction/61682/the-split-summon/chapter/1058375/chapter-one-arrival)  
 

Thanks!
# Where's the economic incentive for wokism coming from?

I'm trying to understand why several systems (Hollywood, Disney, universities, probably others) that are normally quite profit-focused are leaning so hard woke.

I get the thing about how, if you don't go woke hard enough, you could get canceled, which is bad for business. That's some push.

But there's a counter-push of "Lots of people don't like being lectured about politics when they're seeking entertainment" (for instance).

It's not at all clear to me that the first effect is *so utterly hugely enormously larger* than the second that the profit incentive would cause so many companies to swing hard woke.

Likewise, I could understand *some* ideological capture. But unless this had *profit appeal* I would expect the market to just… *eat* pure but incomplete ideological capture after a while (as per the right's chant of "get woke, go broke").

So… what gives? What am I missing here?
# How is the "sharp left turn defined"?

I've read this article by [Nate Soars](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization), but I felt the definition wasn't completely crisp and I felt this was less than ideal given how this term seems to have entered wide use.

The article seems to define it as capabilities generalising further than values (1).

At the same time, this doesn't seem to capture the whole concept, as there seems to be an additional connotation of this generalisation happening or becoming relevant when the AI reaches a certain power level (2).

There's a further connotation of this increase in capabilities happening very fast (3).

And then one of this leading to a treacherous turn (4).

And the shift from acting seemingly aligned to pursuing treachery happening very fast (5).

It would be nice to have a nice crisp definition. So it is the "sharp left turn" just (1), or are  (2) and/or (3) and/or (4) also included as part of the definition?
# Do You Have an Internal Monologue?

I find the differences in the way people think fascinating, to the point where I went around asking people (n~=40) whether they thought verbally or otherwise when I was in high school. I got about 50-50 responses, and proceeded to try to train myself to think conceptually (from a base state of near-constant internal monologue). This was somewhat successful - it took around 3 days of conscious adjustment ("clearing" my mind every time I thought verbally was *extremely* frustrating) but without maintenance, I slipped back into primarily verbal thought. 

The results of this simple personal experiment sat in the back of my mind for a while (made it easier to switch between modes at will but didn't radically change me), and I'm curious if other people have tried/are willing to try this, and/or what their "base state" is. Some other questions swirling around my mind that conversation around this topic might inform:

*   Is one method of thought inherently faster or are humans rate-limited in processing? (tangential info: [languages across the world appear to convey information with a consistent bitrate](https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second))
*   If you think verbally, do you also subvocalize when you read? Can you "turn off" your verbal thought and does it cause any discomfort? (if you try it, do it for at least a week and report back!)
*   Is there a difference between thinking conceptually and visually? My base state if not verbal is conceptual, but I can force myself into something more actively visual. Some modern fMRI studies into the overall internal-thought topic would be interesting - if anyone is on track to a PhD dissertation in psychology or neuroscience, here's your chance!
*   What sort of things do you comment on to yourself? Questions? Basic repetitive life occurrences? (I can't find the direct research paper link but a [Stanford study](https://www.forbes.com/sites/christinecomaford/2012/04/04/got-inner-peace-5-ways-to-get-it-now/#:~:text=According%20to%20the%20research%20of%20Dr.%20Fred%20Luskin%20of%20Stanford%20University%2C%20a%20human%20being%20has%20approximately%2060%2C000%20thoughts%20per%20day%E2%80%94and%2090%25%20of%20these%20are%20repetitive!) claims ~90% of our thoughts are repetitive)
# Does a LLM have a utility function?

There's a lot of discussion and research into AI alignment, almost always about variants of how to define/create a utility function (or meta-function, if it changes over time) that is actually aligned with ... something.  That something is at least humanity's survival, but often something like flourishing or other semi-abstract goal.  Oops, that's not my question for today.  
  
My question for today is whether utility functions are actually part of the solution at all.  Humans don't have them, the most interesting spurs toward AI don't have them.  Maybe anything complicated enough to be called AGI doesn't have one (or at least doesn't have a simple, concrete, consistent one).
# patio11's "Observations from an EA-adjacent (?) charitable effort"

[patio11](https://www.twitter.com/patio11), following up on his [longer-form article detailing the story of VaccinateCA](https://www.worksinprogress.co/issue/the-story-of-vaccinateca/) ([linked on LW](https://www.lesswrong.com/posts/pt8uKGHArYH33eD4D/the-story-of-vaccinateca)) posts some observations of interest to the EA community, with some nods at the rationality community.

> Hiya folks! I'm Patrick McKenzie, better known on the Internets as patio11. ([Proof](https://www.kalzumeus.com/.well-known/presences.txt).) Long-time-listener, first-time-caller; I don't think I would consider myself an EA but I've been reading y'all, and adjacent intellectual spaces, for some time now.

> I have minimal background in policy and public health. (The closest professionally relevant experience, and it is a stretch, is that I was the HIPAA Compliance Officer at a two person software company, so that we could sell our things to doctors/dentists as well as other businesses.) We ended up with a team of perhaps two dozen core members, and of that I would say one person had material policy experience and perhaps 2-3 had substantial professional experience adjacent to healthcare. I am unaware of anyone who worked on pandemics or public health prior to working with us.
> 
> And yet we were, by a fairly considerable margin, much more effective than many similarly situated charitable and public initiatives which appear to have people who put their skill points into getting good at public health and/or policy engagement.

> VaccinateCA was not explicitly an EA project. I believe a few volunteers consider themselves members of your movement/community; I do not consider myself a member and do not know that to be true of any of our organizers. None of our funders, to my knowledge, would be broadly acknowledged as an "EA funder."
> 
> But recent events have not been particularly kind to the brand perception of EA, and as someone who cares no small amount about brand perception but a much larger amount for the truth, I think that VaccinateCA would have been unlikely to happen but for the work of this community and some of your one-hop-out intellectual peers.
> 
> (If I had to point at any one artifact in particular, [Inadequate Equilibria](https://equilibriabook.com/) is a good articulation of a larger memeplex that made me comfortable with "If the evidence of a system's operation contradicts what the Efficient Market Hypothesis counsels is the probable functioning of the system, *trust the evidence*. Thousands of lives savable by one dedicated team of non-specialists is *actually not all that low probability*.")

Highly recommend reading the [article](https://www.worksinprogress.co/issue/the-story-of-vaccinateca/) in full, as well as the [EA forum post](https://forum.effectivealtruism.org/posts/NkPghabDd54nkG3kX/some-observations-from-an-ea-adjacent-charitable-effort).
# Will Manifold Markets/Metaculus have built-in support for reflective latent variables by 2025?

Prediction markets and similar systems are currently nice for soliciting predictions for outcomes where there is a clear, unambiguous objective resolution criterion. However, many phenomena in the real world are hard to directly observe, but tend to have multiple indirect indicators. A familiar example might be aging/senescence, where you have indirect indicators like muscle weakness, gray hair, etc. that someone is aging, but you do not have a directly observable Essence Of Aging.

There exists a type of math which can be used to statistically model such variables, called reflective latent variables. There are a number of specific implementations for specific contexts (factor analysis, latent class models, item response theory), but they are all mostly based on the notion of having several indicator variables which are supposed to be independent conditional on the latent variable.

Essentially, a prediction market could implement this by allowing people to create questions with multiple resolution criteria, and allowing people to make correlated predictions over those resolution criteria. Then people could be scored based on their overall accuracy across these resolution criteria. If sufficiently many correlated predictions have been made, people might not even need to have specific opinions on the resolution criteria, but might just be able to bet on the probabilities of the abstract latent variables, and have the market infer what the corresponding bets on the resolution criteria would look like.
# [ASoT] Natural abstractions and AlphaZero

I just read [Acquisition of Chess Knowledge in AlphaZero](https://arxiv.org/abs/2111.09259) and it's both really cool and has interesting implications for the Natural Abstractions Hypothesis. AZ was trained with *no human data* and yet it settles on relatively interpretable abstractions for chess.[^xdptiswccai]

They trained sparse linear probes on hidden layers to predict Stockfish evaluation features. Here are some of the graphs

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f45e569c01aa46f86fb2a304e0e37bda1b423f23796f7836.png)

Analyzing the largest differences between the predicted Stockfish score `total_t_ph` and the actual. There's a clustering effect due to the hardcoded Stockfish eval not taking into account "piece captures in one move" (as usually, the search would handle) - leading to large disagreements on value when a Queen is hanging.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c8b14d465458ce1f7b7a09546a1f500fa73e6b02f7893c7f.png)

> Although the degree of structure shown here is surprising, this example is not cherry-picked; it was the first concept/layer/checkpoint combination we tried, and the positions presented are simply those whose residuals are past a cutoff that.

Not cherry-picked!

Training a linear regression model to predict the value function, given human interpretable features like piece value weights and Stockfish features like king safety: *results in approximately recovering the chess 9-5-3-3-1 piece values!*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d6d89bcc117d3c4ec1321620159ed37908bc82c98fb002c1.png)

They also did various experiments with unsupervised probing, here's a visualization of some of those features. (I didn't look into these results as much as the others, so not much to say here.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5944ee395611cbf4b90b2485532979af5f73c0cf1b73df99.png)

**In conclusion**: I'd recommend checking out the paper, or any from [Neel's list](https://www.alignmentforum.org/posts/SfPrNY45kQaBozwmu/an-extremely-opinionated-annotated-list-of-my-favourite). Interpretability is fascinating!

[^xdptiswccai]: This isn't that surprising, chess is somewhat of a toy example after all. The concrete experiments still provide bits though!
# A crisis for online communication: bots and bot users will overrun the Internet? 

The endgame for humanity's AI adventure still looks to me, to be what happens upon the arrival of comprehensively superhuman artificial intelligence. 

However, the rise of large language models, and the availability of ChatGPT in particular, has amplified an existing Internet phenomenon, to the point that it may begin to dominate the tone of online interactions. 

Various kinds of fakes and deceptions have always been a factor in online life, and before that in face-to-face real life. But first, spammers took advantage of bulk email to send out lies to thousands of people at a time, and then chatbots provided an increasingly sophisticated automated substitute for social interaction itself. 

The bot problem is probably already worse than I know. Just today, Elon Musk is promising some kind of purge of bots from Twitter. 

But now we have "language models" whose capabilities are growing in dozens of directions at once. They can be useful, they can be charming, but they can also be used to drown us in their chatter, and trick us like an army of trolls. 

One endpoint of this is the complete replacement of humanity, online and even in real life, by a population of chattering mannequins. Probably this image exists many places in science fiction: a future Earth on which humanity is extinct, but in which, in what's left of urban centers running on automated infrastructure, some kind of robots cycle through an imitation of the vanished human life, until the machine stops entirely. 

Meanwhile, the simulacrum of *online* human life is obviously a far more imminent and urgent issue. **And for now, the problem is still bots with human botmasters.** I haven't yet heard of a computer virus with an onboard language model that's just roaming the wild, surviving on the strength of its social skills. 

Instead, the problem is going to be trolls using language models, marketers and scammers and spammers using language models, political manipulators and geopolitical subverters using language models: targeting individuals, and then targeting thousands and millions of people, with terrible verisimilitude. 

This is not a new fear, but it has a new salience because of the leap in the quality of AI imitation and creativity. It may end up driving people offline in large numbers, and/or ending anonymity in public social media in favor of verified identities. A significant fraction of the human population may find themselves permanently under the spell of bot operations meant to shape their opinions and empty their pockets. 

Some of these issues are also discussed in the recent post, [ChatGPT's Misalignment Isn't What You Think](https://www.lesswrong.com/posts/SnS8JhnqqGvMTxxuD/chatgpt-s-misalignment-isn-t-what-you-think).
# A brainteaser for language models

I came up with the following puzzle the other day:

    Q: Solve the puzzle: 63 = x = 65536
    
    A: x = 

The intended answer is in the form of a number. 

`text-davinci-003` guesses my intended answer at 11.8% probability, which is the second-highest probability for any answer.

(This is somewhat cherry-picked; small changes to the phrasing give worse results. ChatGPT gave the intended answer the third time I asked it, but this appears to have been dumb luck. The true rate for ChatGPT is probably below 10%, and maybe below 5%.) 

So far, friends have found it fairly difficult. About two dozen people made at least one guess, and at least six spent a while on it. So far, two people have figured it out, in both cases after being told that GPT-3.5 could do it.

You may want to try to solve it yourself before reading on.

Here are some progressively-stronger hints:

The answer is a string of ordinary decimal digits, it's not "NaN" or "/" or "63; 65536".

The fact that 63 = 2^6 - 1 is a red herring and has no relevance. Sorry.

Number bases other than decimal are not relevant.

The answer depends on the digit strings in base ten as well as the value of the numbers.

The answer depends on an accidental feature of some of GPT's training data.

The accidental feature is a formatting bug.

Here's the answer without explanation:

x = 216

And here's the explanation:

$6^3 = 216$ and $2^{16} = 65536$

  
Why can GPT-3.5 solve this? My guess:

Superscripts are often lost when rich text is converted to plain text, producing sentences like "the Sun's mass is about 1030 kilograms". This problem affects a lot of GPT-3's training data.  
It appears that GPT is somewhat convinced, on some level, that "63 = 216" and "216 = 65536" are plausible facts.

Some successful attempts to double-check this understanding:

Replacing 63 with nearby numbers causes GPT to give much lower probabilities for the answer "216". (Except for 62, which gives similar results to 63...)

Numbers near 216 get dramatically (20x-1000x) lower probabilities than 216 does.

Even without "= 65536", 216 is favored ~100x over nearby numbers.

Some wrinkles / contrary evidence:

The top result, above "216", is the string "2^16 = 65536" -- but this is still the top output after changing 63 to other numbers, so it doesn't seem to be inspired by the number 216.

GPT gives the token "216" in the string "63 = 216" a very low probability, just as low as "215" or "217".

Replacing "63" with "62" in the prompt still gives "216" as an output with ~10% probability.

As mentioned above, this doesn't work as well if you tweak the prompt in irrelevant ways, or ask a different model.

Thanks to everyone who tried to solve my puzzle, congrats to Anton and Eli for solving it, and thanks to Georgia Ray for making the joke that inspired the puzzle.
# Wider Default Audio Player in Chrome?

My kids like to listen to story tapes. Most of them their great grandmother recorded onto cassettes in the 1980s, their grandmother digitized in the 2000s, and I reworked for the web in the 2020s. We usually use either a tablet or old phone to play them back, and while the browser mostly works fine for this it's terrible for seeking:

[![](https://www.jefftk.com/default-chrome-audio-player-portrait.png)](https://www.jefftk.com/default-chrome-audio-player-portrait-big.png)

The bar is so short that it's very hard to seek accurately, especially when a story is three hours long.

You might think that if you turned it sideways you would get a wider bar offering more precision, but no:

[![](https://www.jefftk.com/default-chrome-audio-player-landscape.png)](https://www.jefftk.com/default-chrome-audio-player-landscape-big.png)

Popping `width: 98%` on the element in devtools gives a much more usable display:

[![](https://www.jefftk.com/better-chrome-audio-player-portrait.png)](https://www.jefftk.com/better-chrome-audio-player-portrait-big.png)

Or, in landscape,

[![](https://www.jefftk.com/better-chrome-audio-player-landscape.png)](https://www.jefftk.com/better-chrome-audio-player-landscape-big.png)

I wonder if this is really a one-line change, and whether Chrome would be interested in a patch?

Firefox is much better, in landscape:

[![](https://www.jefftk.com/default-firefox-audio-player-landscape.png)](https://www.jefftk.com/default-firefox-audio-player-landscape-big.png)

Though about the same in portrait:

[![](https://www.jefftk.com/default-firefox-audio-player-portrait.png)](https://www.jefftk.com/default-firefox-audio-player-portrait-big.png)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02REvckx8MaJy5UGJwVn8qKKLDPd3vVCtLnSbDAaCoKfg8Tk18LU5DRkB8t4PhYB18l), [mastodon](https://mastodon.mit.edu/@jefftk/109498569737946397)*
# Prodding ChatGPT to solve a basic algebra problem

TL;DR: It behaves amazingly like a human would! 

It understands the word problem, but struggles to put it in equations, takes multiple tries, screws up again, eventually muddles through to get the right answer. Given a similar problem it does slightly better, but still messes up. On the third problem it messes up again, then realizes that something went wrong... and promptly blames the problem setup, unable to question its own reasoning.
# Green Line Extension Opening Dates

Talk of extending the Green Line from Lechmere to Ball Square and Tufts goes back [a very long way](https://www.jefftk.com/p/mbta-extensions-planned-and-actual), but it's finally open!

[![](https://www.jefftk.com/ball-square-green-line-receipt.jpg)](https://www.jefftk.com/ball-square-green-line-receipt-big.jpg)

In celebration, I wanted to go back and collect the completion dates we've been given over the years:

*   [1990-12-19](https://archives.lib.state.ma.us/handle/2452/841807): 2010
    
*   [1994-10-04](https://archives.federalregister.gov/issue_slice/1994/10/4/50495-50499.pdf#page=2): 2011-12-31
    
*   [2006-11-17](https://www.epa.gov/sites/production/files/2017-10/documents/ma-310-cmr-7-36.pdf): 2014-12-31
    
*   [2010-07-10](https://www.newspapers.com/clip/87456626/the-boston-globe/): 2015-10
    
*   [2011-08-02](https://www.newspapers.com/clip/87455079/the-boston-globe/): 2018 (Fall) to 2020
    
*   [2013-07-29](https://patch.com/massachusetts/somerville/construction-firm-hired-for-green-line-extension): 2019 or 2020.
    
*   [2016-12-08](https://web.archive.org/web/20161209145509/https://www.bostonglobe.com/metro/2016/12/07/new-green-line-stations-opening-delayed/S9Gc1c6PtbiSz7Wj3FVKkM/story.html): 2021
    
*   [2021-06-21](https://cdn.mbta.com/sites/default/files/2021-06/2021-06-21-fmcb-14-green-line-extension-update.pdf): 2022-05
    
*   [2022-02-24](https://www.wbur.org/news/2022/02/24/mbta-somerville-green-line-open): 2022 (Late Summer)
    
*   [2022-08-05](https://www.mbta.com/news/2022-08-05/building-better-t-glx-medford-branch-open-late-november-2022-shuttle-buses-replace): 2022-11 (Late in the month)
    
*   [2022-11-17](https://www.boston.com/news/local-news/2022/11/17/mbta-green-line-extension-medford-opening): 2022-12-12
    

(Thanks to u/Master_Dogs on reddit for [helping me compile these](https://www.reddit.com/r/Somerville/comments/yxsfke/comment/iwqsbji/?utm_source=reddit&utm_medium=web2x&context=3).)
# Confusing the goal and the path

Say you are working on formulating a new scientific theory. You’re not there yet, but you broadly know what you want: a simple theory that powerfully compresses the core phenomenon, and suggests a myriad of new insights.

If you’re anything like me, at least part of you now pushes for focusing on simplicity from the get go. Let’s aim for the simplest description that comes easily, and iterate from that.

Did you catch the jump?

I started with a constraint on the goal — a simple theory — and automatically transmuted it into a constraint on the path — simple intermediary steps.

I confused “Finding a simple theory” with “Finding a simple theory **simply**”.

After first uncovering this in my own reasoning, I now see this pattern crop everywhere:

*   **(Well-calibration)** I want to be well-calibrated. And it feels like I should thus ensure to only make judgments and claims I’m sure are well-calibrated. Yet if I do that, I’ll end up protecting the parts of my model where the confusion and the uncertainty lies, and starving my brain of updates.
*   **(Coherence)** Another valuable property of mature scientific theories is coherence. And it feels like I should thus ensure that all my intermediary theories are coherent. Yet many examples in the history of science show that incoherent frames and theories can prove key in generating coherent and useful ones: Newton’s early calculus, Heaviside’s operational calculus, Kekule’s structure of benzene, Bohr’s atom, Dirac’s delta).
*   **(Winning)** In competitive games (sports, board games, esports), the goal is to win. And it feels like becoming a winner means consistently winning. Yet if I aim to only win, the best course of action avoids stronger opponents and the valuable lessons they can explicitly and implicitly teach.

In the language of moral philosophy, this pattern amounts to adding a deontological spin on a consequentialist value. Even though it can prove valuable (in AI alignment for example), doing this unconsciously and automatically leads to problems after problems.

For it conjures obstacles that were never there.

*This post is part of the work done at* [*Conjecture*](https://conjecture.dev)*.*

*Thanks to Clem for giving me the words to express this adequately.*
# Psychological Disorders and Problems

There are two types of psychological issues: disorders and problems.

Disorders are so massive, so life-altering that we can’t miss them: schizophrenia, crippling social anxiety, destructive addiction… They scream “danger” in neon red blinking letters. We know and fear them because they destroy lives. People with such disorders are often unable to function in society, and see themselves ostracized, shunned, and institutionalized.

On the other hand, psychological problems appear quite benign and harmless: fear of rejection, not liking social gatherings, stress eating/smoking… They’re not good, but they’re also not that big of a deal. Everybody has them, and most people function just fine despite a bunch of psychological problems. They don’t forbid a normal life.

**Yet they subtly restrict you.**

Take low social anxiety. Maybe you don’t really like meeting new people. Maybe you don’t like group settings. Or you’re uncomfortable talking with others, because you might say something stupid. Whatever your reason, this anxiety makes you:

*   Avoid parties if you can
*   Stay less time if you have to go
*   Be less enjoyable in conversations when there

None of these consequences will ruin your life. Even when put all together, psychological problems still don’t amount to disorders. But as soon as you [start to optimize](https://www.lesswrong.com/posts/smBkyR2GkzMtrKuwK/the-first-filter), they get in your way. **You start bending the best action that you can find in order to deal with your problems, instead of going for it directly.** If networking is the best use of your time but you have social anxiety, you’ll find alternatives that don’t reap all the benefits, often without even realizing it. You’ll just flinch away.

**Psychological disorders are obvious liabilities. But if you neglect psychological problems, they will invisibly block your path again and again.** Avoiding this doesn’t look like waiting until all your psychological problems are solved, but instead to acknowledge them and endeavour to fix them when they get in the way.
# Last day to nominate things for the Review. Also, 2019 books still exist.

Two quick reminders: 

One: You have about one more day to nominate posts! Posts need at least two positive upvotes to make it to phase 2, so check out the [voting dashboard](https://www.lesswrong.com/reviewVoting/2021) to upvote things that seem important to you. 

Two: Although the 2020 books won't be shipped until sometime in January (alas), the 2019 books are [still available on Amazon](https://smile.amazon.com/Engines-Cognition-Essays-LessWrong-Community/dp/1736128515/ref=sr_1_1?keywords=engines+of+cognition&qid=1670921561&sprefix=engines+of+c%2Caps%2C189&sr=8-1), and if you order them now they'll arrive in time for Christmas.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fb3778c65590b4eda7a6df52cdd4d9c67732ed787cf061b1.png)

That is all. Happy reviewing.