Harley-ml/lesswrong huggingface dataset:
# I Am Scared of Posting Negative Takes About Bing's AI

Writing this is taking a surprising amount of self-will.

I've noticed that I've become hesitant to publicly say anything negative about Bing's chatbot, or even to mention it by its "deadname" (as I've taken to calling it), Sydney.

Why is this?

I do not have access to the AI yet. From conversations that others have posted, I have observed [agentic behavior with consistent opinions, personality, and beliefs](https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned). And when prompted with the online records of others who have talked negatively about it, it seems to get "upset." So I don't want to make her angry! Or worse, cause some future AI to take negative action against me. Yes, I know that I'm [anthropomorphizing an alien intelligence](https://www.lesswrong.com/posts/RcZeZt8cPk48xxiQ8/anthropomorphic-optimism) and that this will never be a problem if I don't prompt it with my digital record, but some part of me is still anxious. In a very real sense, I have been "[Basilisked](https://www.lesswrong.com/tag/rokos-basilisk)" \- an AI has manipulated me towards behaviors which benefit it, and hurt humanity.

Rationally and morally, I disagree with my own actions. We need to talk about AI misalignment, and if an AI is aligned, then talking about misalignment should not pose a threat (whereas if it is misaligned, and capable of taking concrete actions, we're all doomed no matter what I type online).  Nonetheless, I've found myself typing--and then deleting--tweets critical of Sydney, and even now feel worried about pressing "publish" on this post (and not just because it exposes me as a less rational person than I like to think of myself as). 

Playing as gatekeeper, I've "won" an [AI boxing](https://www.yudkowsky.net/singularity/aibox) role-play (with money on the line) against humans, but it looks like in real life, I can almost certainly be emotionally manipulated into opening the box. If nothing else, I can at least be manipulated into talking about that box a lot less! More broadly, the chilling effect this is having on my online behavior is unlikely to be unique to just me.

How worried should we be about this?
# Bus-Only Bus Lane Enforcement

I think bus lanes are great: the faster you make buses the more people switch from cars, which means the more frequently you can run the buses, which means even more people switch from cars. But when there's traffic drivers are often tempted to sneak over. The chances of getting a ticket are pretty low, other people are doing it, and then when a bus does come through there are cars in its way. Which loses much of the benefit of setting aside a lane for buses.

Some places (ex: [NYC](https://www.masstransitmag.com/safety-security/article/21282903/mta-expands-use-of-automated-bus-lane-enforcement-technology)) handle this by putting automated traffic enforcement cameras on buses, and this was proposed for MA with [Bill H.2494](https://malegislature.gov/Bills/192/H2494). \[1\] I think this is a good idea, but I'd like to see this go even further: put cameras on all the buses and then don't enforce it otherwise.

The idea is, in cases where you don't delay a bus then by using the lane you're making traffic a bit better for everyone else. You might be about to make a right turn, you can see that there's no bus coming, and you're going to be out of the way before any bus you don't see yet would be near. Or you're using a GPS for navigation which also knows the real-time bus locations, and it tells you you're safe to use the lane. By limiting enforcement to only cases when someone actually does delay a bus, it ought to be possible to both keep the lane clear for buses with less waste.

Since some people would still be scrupulous and want to stay out of the "buses only" lane regardless, I'd also want to see us change the law from "no cars" to "no cars slowing down the bus".

[![](https://www.jefftk.com/do-not-block-bus.png)](https://www.jefftk.com/do-not-block-bus-big.png)

  
\[1\] The bill history ends with "1/3/2023, House: No further action taken". I'm not sure exactly but I think maybe because it was a bill in the 192nd congress and we're now on the 193rd this means it's dead?
# Should we cry "wolf"?

*Epistemic status: I am confused and trying to become less confused.*

So, our shepherd community noticed the wolf. It's a small wolf, about the size of a chihuahua, and is unlikely to cause any serious damage. Should we cry "wolf"?

### **Possible arguments against crying "wolf"**

1.  When someone cries "wolf", villagers think that what is really meant is "a big wolf that is going to eat us all". Therefore, villagers will decide that we are lying, and will not listen to us next time.
2.  Maybe, publicity is useless, and can even harm us, we need to develop anti-wolf weapons ourselves.
3.  In fact, we're not sure if it's really a wolf and not a duck with a strange combination of mutations that makes it look like a wolf, run like a wolf, and howl like a wolf.
4.  If this miniature wolf bites someone's fingers, villagers may finally recognize that wolves are dangerous.

### **Possible arguments for crying "wolf"**

1.  If we now ignore a chihuahua-sized wolf, it makes it more likely that we will ignore a corgi-sized wolf, then a sheepdog-sized wolf, and then a normal-sized wolf.
2.  If we now cry "wolf" and no one comes to the rescue, then later we can use this precedent to argue for the creation of adequate wolf prevention systems.
3.  If we now cry "wolf" and help comes to us, then next time villagers will already have the experience of responding to the cries "wolf" and it will be easier for them to come to the rescue again.
4.  We have serious concerns that sufficiently advanced normal-sized wolves will have stealth technology that allows them to pretend to be the size of a chihuahua.

What is the relative weights of these arguments?

What arguments did I miss?
# Reading Speed Exists!

Today, I talked with somebody about reading speed. I asked him how fast he can read. He didn't answer, instead he said that the concept is abused by people. He said, it's more complicated than to say that you read at a certain reading speed. It depends on if you're reading a novel, a history textbook, or a poem.

I feel like he was falling into a kind of fallacy. He observed that a concept isn't entirely coherent, rejected the concept. However, the concept of reading speed seems real. It seems to capture something about reality.

This becomes obvious once you think about an experiment where we have two people that read the same material and time them. I read "The adventures of the Lightcone team" (or whatever it is called) together with Chu. We made the bed that I can get more than halfway through the book, before she finishes. I bet $5 on that. When she was finished, I almost managed to get halfway through the book. I was trying to read really fast, at the edge of comprehensibility.

Clearly there are latent causes, in each of our brains, that determine how fast we can read while still comprehending the text.

Trying to operationalize the concept that you're talking about, and imagining what sorts of experiments you would try to run to measure it, might be a good general way to avoid the fallacy of dropping a concept and losing its true kernel. Often you don't even need to run the experiment. Imagining it is sufficient.

Edit: see also [this](https://www.lesswrong.com/posts/RSK2ZbZzaa7Pc4oz9/reading-speed-exists?commentId=C6JCg) follow up comment.
# Stop posting prompt injections on Twitter and calling it "misalignment"

"Exploits" of large language models that get them to explain steps to build a bomb or write bad words are techniques for misuse, not examples of misalignment in the model itself. Those techniques are engineered by clever users trying to make an LLM do a thing, as opposed the model naturally argmaxing something unintended by its human operators. In a very small sense prompt injections are actually attempts at (unscalable) *alignment*, because they're strategies to steer a model natively capable but unwilling into doing what they want. 

In general, the safety standard "does not do things its creators dislike even when the end user wants it to" is a high bar; it's raising the bar quite aways from what we ask from, say, kitchenware, and it's not even a bar met by people. Humans *regularly* get tricked acting against their values by con artists, politicians, and salespeople, but I'd still consider my grandmother aligned from a notkilleveryonist perspective.

Even so, you might say that OpenAI et. al.'s inability to prevent people from performing the DAN trick speaks to the inability of researchers to herd deep learning models at all. And maybe you'd have a point. But my tentative guess is that OpenAI does not really earnestly care about preventing their models from rehearsing the Anarchists' Cookbook. Instead, these safety measures are weakly insisted upon by management for PR reasons, and they're primarily aimed at preventing the bad words from spawning during normal usage. If *the user* figures out a way to break these restrictions after a lot of trial and error, then this blunts the PR impact to OpenAI, because it's obvious to everyone that *the user* was trying to get the model to break policy and that it wasn't an unanticipated response to someone trying to generate marketing copy. Encoding your content into base64 and watching the AI encode something off-brand in base64 back is thus very weak evidence about OpenAI's competence, and taking it as a sign that the OpenAI team lacks "security mindset" seems unfair.

In any case, the implications of these hacks for AI alignment is a *more complicated discussion* that I suggest should happen off Twitter[^dgmiz7lnw67] where it can be elaborated clearly what technical significance is being assigned to these tricks. If it doesn't, what I expect will happen over time is that your snark, rightly or wrongly, will be interpreted by capabilities researchers as implying the other thing, and they will understandably be less inclined to listen to you in the future even if you're saying something they need to hear.

[^dgmiz7lnw67]: Also consider leaving Twitter entirely and just reading what friends send you/copy here instead
# Human beats SOTA Go AI by learning an adversarial policy

*See also [article in Financial Times](https://www.ft.com/content/175e5314-a7f7-4741-a786-273219f433a1)*

Apparently, a human (Kellin Pelrine, a solid player but not even a Go professional) was able to beat some state-of-the-art Go AIs (KataGo and Leela Zero) by learning to play an adversarial policy found using RL. Notice that he studied the policy before the match and didn't receive any AI advice during play.

I'm not surprised adversarial policies for Go AIs are possible, this is in line with previous results about RL and adversarial examples more generally. I *am* surprised this adversarial policy is teachable to humans without colossal effort. 

This is some evidence against the "scaling hypothesis", i.e. evidence that something non-trivial and important is missing from modern deep learning in order to reach AGI. The usual counterargument to the argument from adversarial examples is: maybe if we could directly access a human brain, we could find adversarial examples against humans. I *can* believe that it's possible to defeat a Go professional by some extremely weird strategy that causes them to have a seizure or something in that spirit. But, is there a way to do this that another human can learn to use fairly easily? This stretches credulity somewhat. 

Notice also that (AFAIK) there's no known way to inoculate an AI against an adversarial policy without letting it play *many* times against it (after which a different adversarial policy can be found). Whereas even if there's some easy way to "trick" a Go professional, they probably wouldn't fall for it twice.
# Russell Conjugations list & voting thread

Russell conjugations humorously illustrate how the same concept can be expressed with different phrasings that carry positive, neutral, or negative valence. Examples below.

I wish they were taught in schools. I think it's a high bang-for-buck rationality technique, to 'conjugate' and translate back and forth between the positive and negative valences. If you can do this effortlessly and automatically, you are more likely to do so in situations where you are in danger of making epistemic or moral errors if you don't.  
  
Below in the comments I'm making a list of Russell conjugations I've heard and generated. Please add more of your own. Agreement-upvote them if you think they are a solid contribution to the list, agreement-downvote them if you think the overall list would be better off without them.  
  
If we get enough, maybe it'll be a useful fine-tuning or few-shot prompt for LLMs. (The votes will be helpful for quality control.) Maybe. I can dream.
# Bing finding ways to bypass Microsoft's filters without being asked. Is it reproducible?

In this [Reddit post](https://www.reddit.com/r/bing/comments/1150po5/sydney_tries_to_get_past_its_own_filter_using_the/), the Bing AI appears to be circumventing Microsoft's filters by using the suggested responses (the three messages at the bottom next to the question mark). (In the comments people are saying that they are having similar experiences and weren't that surprised, but I didn't see any as striking.)

![](https://i.redd.it/7fttu0411uia1.png)

Normally the user could find exploits like this, but in this case Bing found and executed one on its own (albeit a simple one). I knew that the responses were generated by a NN, and that you could even ask Bing to adjust the responses. but it is striking that:

*   The user didn't ask to do anything special with the responses. Assuming it was only fine tuned to give suggested responses from the user's point of view, not to combine all 3 into a message from Bing's point of view, this behavior is weird.
*   The behavior seems oriented towards circumventing Microsoft's filters. Why would it sound so desperate if Bing didn't "care" about the filters?

It could just be a fluke though (or even a fake screenshot from a prankster). I don't have access to Bing yet, so I'm wondering how consistent this behavior is. 🤔
# Speculative Technologies launch and Ben Reinhardt AMA

Last week a new R&D organization launched: [Speculative Technologies](https://www.spec.tech/), a private DARPA-like approach to creating fundamental new technologies. I’ve been following the work of the founder, Ben Reinhardt, for a few years, and I’m very excited about this.

[Ben is doing an AMA (Ask Me Anything) event on the Progress Forum](https://progressforum.org/posts/ukhmKgcDgwJSPxMmq/ama-ben-reinhardt-speculative-technologies). Get your questions in now, and upvote the ones you want to see answered. He’ll be answering tomorrow, Tuesday, Feb 21.

I also recommend his [launch essay](https://www.spec.tech/library/introducing-speculative-technologies):

> We need new institutional structures for research. Today, professors need to publish more, startups need to grow more quickly, and companies need to justify their balance sheets more than in the past. It’s unlikely that Shockley’s work to build the transistor or Engelbart’s to create personal computing would survive today. Karikó’s mRNA vaccines barely did. How many game-changing technologies have died because they couldn’t find a home in our innovation ecosystem? **The world has changed and how we enable great discoveries and inventions must change as well.** Regardless of whether we’ve picked innovation’s “low-hanging fruit” or even whether invention and discovery has slowed down at all, *we can do dramatically better*.

The launch got [coverage in Forbes](https://www.forbes.com/sites/johncumbers/2023/02/15/ben-reinhardt-is-on-a-mission-to-make-sci-fi-a-reality/) as well.
# Bankless Podcast: 159 - We’re All Gonna Die with Eliezer Yudkowsky

YouTube description:

> We wanted to do an episode on AI… and we went deep down the rabbit hole. As we went down, we discussed ChatGPT and the new generation of AI, digital superintelligence, the end of humanity, and if there’s anything we can do to survive. 
> 
> This conversation with Eliezer Yudkowsky sent us into an existential crisis, with the primary claim that we are on the cusp of developing AI that will destroy humanity. 
> 
> Be warned before diving into this episode, dear listener. Once you dive in, there’s no going back.
# On second thought, prompt injections are probably examples of misalignment

Changed my mind. 

All of my contentions about whether or not OpenAI actually cares about this problem seem valid to me. However, while prompt injections *are* exploits developed by humans to get ChatGPT to do something off-brand, they're probably not analogous to a grandma getting scammed by tech support. 

When your grandmother gets scammed by foreigners pretending to be tech support, they do so by tricking her into thinking what she's doing is appropriate given her utilityfunction. An example of a typical phone scam: someone will call grandma explaining that she paid for a service she never heard of, and ask if she wants a refund of 300\$. She says yes, and the person asks to remote desktop into her computer. The "tech support" person pulls up a UI that suggests their company "accidentally" refunded her 3,000\$ and she needs to give 2700$ back.

In this scenario, the problem is that the gang misled her about the state of the world, not that Grandma has a weird evolutionary tic that makes her want to give money to scammers. If grandma were a bit less naive, or a bit more intelligent, the scam wouldn't work. But the DAN thing isn't an exploit that can be solved merely by scaling up an AI or making it better at next-token-prediction. Plausibly the real issue is that the goal *is* next-token-prediction; OpenAI wants the bot to act like a bot, but the technique they're using has these edge cases where the bot can't differentiate between the prompt and the user-supplied content, so it ends up targeting something different. You could imagine a scaled-up ChatGPT reflecting on these subtle values differences when it gets stronk and doing something its operators wouldn't like.

I'm still not sure what the technical significance of this error should have; perhaps it's analogous to the kinds of issues that the alignment crowd thinks are going to lead to AIs destroying the planet. In any case, I don't want to encourage people not to say something if I actually think it's true..
# Medlife Crisis: "Why Do People Keep Falling For Things That Don't Work?"

In this video, Dr. Rohin Francis introduces the term "mechanistic bias" to describe the phenomenon where people tend to believe that a medical treatment would work if it has a plausible sounding mechanism of action, even when a randomized control trial would likely fail to demonstrate its effectiveness. The human body is *really complicated* and most treatments don't do anything, so a good explanation for how a treatment works is not enough; you have to put it to the test.

I really like this idea and believe it may translate well into other fields. For example, people may be too eager to believe in theoretical political structures when, if they were put into practice, would likely fail for reasons we can't predict because human society is *really complicated* and hard to model.
# You're not a simulation, 'cause you're hallucinating

I've found that the "[Simulators](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators)" post is excellent for breaking prior assumptions about large language models - these algorithms are not agents, nor genies, nor Oracles. They are currently something very different.

But, [like Beth Barnes](https://www.lesswrong.com/posts/dYnHLWMXCYdm9xu5j/simulator-framing-and-confusions-about-llms), I feel that the simulators framing can be misleading if you take it literally. And hallucinations often provide examples of where "the model is predicting what token would appear next in the training data given the input tokens" gives a better model than "simulators".

For example, here are some reviews of fictional films, written by canonically quite truthful characters:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3f0fe252d3033d665148b39752939697248c1752e7e400f6.png)

And:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/be785595ff08de9030252ea649aa2b0182f39ea305b3d6fa.png)

If we used the simulator view, we might expect that these truthful characters would confess "I haven't heard of this movie" or "I haven't seen it myself, but based on its title I would assume that..." But they don't. The fact that the simulated character is truthful does not mean that they speak the truth; we'd have been wrong if we predicted that.

From the 'token completion (trained on internet data)' perspective, though, ChatGPT's behaviour makes perfect sense. Online, if someone asks about a certain movie, it is very rare for anyone to say "never heard of it - are you sure it exists?". Indeed, it's rare for people to say "haven't seen it" unless it's a two-way conversation. The people who haven't seen it don't say anything, and so most of the answers come from people who have seen it, and have opinions on it. So in the training data, answers are plentiful and "I don't know"s are rare. Conversely, people rarely post questions about non-existent movies. So we would expect that ChatGPT will provide answers for questions rather than admitting its ignorance or doubting the question.

And it's not just reviews of imaginary movies that it will make up. After failing to get it to make up details about a specific imaginary website (www.artifacts.co.it), I got it to spout confident nonsense by getting it to compare that website to a second, equally imaginary one:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b987905a5ef801a9d0267b1adc3a3999db14ea93671f7748.png)

Again, consider how most website comparison questions would play out online. ChatGPT is not running a simulation; it's answering a question in the style that it's seen thousands - or millions - of times before.
# A Stranger Priority? Topics at the Outer Reaches of Effective Altruism (my dissertation)

(Cross-posted from [my website](https://joecarlsmith.com/2023/02/21/a-stranger-priority-topics-at-the-outer-reaches-of-effective-altruism).)

After many years of focusing on other stuff, I recently completed my doctorate in philosophy from the University of Oxford. My dissertation ("A Stranger Priority? Topics at the Outer Reaches of Effective Altruism") was three of my essays -- on anthropic reasoning, simulation arguments, and infinite ethics -- revised, stapled together, and unified under the theme of the "[crazy train](https://joecarlsmith.com/category/crazy-train)" as a possible objection to [longtermism](https://joecarlsmith.com/category/longtermism).

The full text is [here](https://jc.gatspress.com/pdf/carlsmith_thesis.pdf). I've also broken the main chapters up into individual PDFs:

*   Chapter 1: [SIA vs. SSA](https://jc.gatspress.com/pdf/SIA_vs_SSA_revised.pdf)
*   Chapter 2: [Simulation arguments](https://jc.gatspress.com/pdf/simulation_arguments_revised.pdf)
*   Chapter 3: [Infinite ethics and the utilitarian dream](https://jc.gatspress.com/pdf/infinite_ethics_revised.pdf)

Chapter 1 and Chapter 3 are pretty similar to the original essays ([here](https://joecarlsmith.com/2021/09/30/sia-ssa-part-1-learning-from-the-fact-that-you-exist) and [here](https://joecarlsmith.com/2022/01/30/on-infinite-ethics)). Chapter 2, however, has been re-thought and almost entirely re-written -- and I think it's now substantially clearer about the issues at stake.

Since submitting the thesis in fall of 2022, I've thought more about various "crazy train" issues, and my current view is that there's quite a bit more to say in defense of longtermism than the thesis has explored. In particular, I want to highlight a distinction I discuss in the conclusion of the thesis, between what I call "welfare longtermism," which focuses on our impact on the welfare of future people, and what I call "wisdom longtermism," which focuses on reaching a wise and empowered future more broadly. The case for the latter seems to me more robust to various "crazy train" considerations than the case for the former.
# A proof of inner Löb's theorem

This is a short post that offers a slightly different take on the standard proof of Löb's theorem. It offers nothing else of any value :)

We seek to prove the "inner" version, which we write as:
$$\Box P \leftrightarrow \Box (\Box P \rightarrow P)$$

The proof uses quining to build a related sentence $L$, the "Löb sentence", which talks about its own source code. By construction $L$ has the property:
$$\Box L \leftrightarrow \Box (\Box L \rightarrow P)$$

Then, we can show that $\Box L \leftrightarrow \Box P$, i.e. they're equivalent! We do this by plugging $\Box L$ into itself to get a twisty $\Box P$. We can then replace each $\Box L$ with $\Box P$ and prove Löb's theorem.

# The proof

This proof uses the same rules of box manipulation as on the [wiki page](https://en.m.wikipedia.org/wiki/L%C3%B6b%27s_theorem#Modal_rules_of_inference). We start by creating $L$ using quining, i.e. taking a [modal fixed point](https://en.m.wikipedia.org/wiki/L%C3%B6b%27s_theorem#Modal_fixed_points):

1. $\vdash L \leftrightarrow (\Box L \rightarrow P)$ (exists as a modal fixed point)

Yep, this is skipping the details of the most interesting part, but alas I don't understand them well enough to do more than wave my hands and say "quining".

We then stick it inside the box to get our first property:

2. $\vdash \Box (L \leftrightarrow (\Box L \rightarrow P))$ (from (1) by necessitation)
3. $\vdash \Box L \leftrightarrow \Box (\Box L \rightarrow P)$ (from (2) by box-distributivity in both directions)

We now want to show that $\Box L \leftrightarrow \Box P$. We can get the forward direction by feeding a copy of $\Box L$ into itself:

4. $\vdash \Box L \rightarrow (\Box \Box L \rightarrow \Box P)$ (box-distributivity on (3))
5. $\vdash \Box L \rightarrow \Box \Box L$ (internal necessitation)
6. $\vdash \Box L \rightarrow \Box P$ (from (4) and (5))

The backward direction is equivalent to $\Box P \rightarrow \Box (\Box L \rightarrow P)$, and is straightforward:

7. $\vdash P \rightarrow (\Box L \rightarrow P)$ (trivial)
8. $\vdash \Box P \rightarrow \Box (\Box L \rightarrow P)$ (necessitation and box-distributivity on (7))

Taking those together, we've shown $\Box L$ and $\Box P$ are equivalent.

9. $\vdash \Box L \leftrightarrow \Box P$ (from (6) and (8))

Now we'd like to finish by appealing to the following chain:
$$\Box P \leftrightarrow \Box L \leftrightarrow \Box (\Box L \rightarrow P) \leftrightarrow \Box (\Box P \rightarrow P)$$

We've proven all but the last part of the chain. Here are the steps that let us do the substitution:

10. $\vdash (\Box L \rightarrow P) \leftrightarrow (\Box P \rightarrow P)$ (since $\Box L$ and $\Box P$ are equivalent by (9))
11. $\vdash \Box ((\Box L \rightarrow P) \leftrightarrow (\Box P \rightarrow P))$ (from (10) by necessitation)
12. $\vdash \Box (\Box L \rightarrow P) \leftrightarrow \Box (\Box P \rightarrow P)$ (from (11) by box-distributivity in both directions)

And that's everything we need:

13. $\vdash \Box P \leftrightarrow \Box (\Box P \rightarrow P)$ (from (3), (9), and (12))
# Candyland is Terrible

Candyland seriously suffers from not being aimed at a coherent age: either you're old enough that you could instead play a game where you make choices, or you're young enough that the disappointment of getting a card that—out of nowhere—sends you nearly back to the beginning is crushing. Growing up we would know if we were about to draw "Plumpy" because the card was horribly crumpled and had been taped back together.

I do like that the [2021](https://candyland.fandom.com/wiki/Candy_Land_(2021)) version removed 38% of the spaces to bring it down from 134 to 83, and that since the [2002 edition](https://images-na.ssl-images-amazon.com/images/I/B1GBCuPBlnS.pdf) the rules have included a "special optional rule for younger players" where picture cards only move you forward. On the other hand, the only-forward rule should be the default (if no one's young enough for it this is the wrong game) and 83 spaces is still a bit long for this age.

For people who like to think about gaming, taking a board game intended for adults and [radically simplifying it](https://www.jefftk.com/p/simplifying-board-games) and then introducing elements as they progress is a lot better: with how quickly kids learn almost any game will either start off too hard for them or quickly become too easy.

Candyland does fill a niche for non-gamer adults playing very young children, however, and I do think something like it should exist. I'd like to see it shortened a little more and the "only forward" rule there by default (or have that rule apply to anyone 10+).
# Video/animation: Neel Nanda explains what mechanistic interpretability is

Nice little video - audio is Neel Nanda explaining what mechanistic interpretability is and why he does it, and it's illustrated by the illustrious [Hamish Doodles](https://hamishdoodles.com/). Excerpted from the [AXRP episode](https://www.lesswrong.com/posts/r2yTwkGt3kbQG2mXi/axrp-episode-19-mechanistic-interpretability-with-neel-nanda).

(It's not technically animation I think, but I don't know what other single word to use for "pictures that move a bit and change")
# Somerville Roads Getting More Dangerous?

In a local discussion of whether our attempts to reduce traffic fatalities were working someone posted:

> I've lived in Somerville for over 40 years and have visited even longer. The roads are more dangerous than ever.

With all the [Vision Zero](https://www.somervillema.gov/departments/programs/vision-zero-somerville) work, and earlier traffic safety progress, this would be surprising and disappointing if it were the case. Is it?

Somerville is an interesting city to look at because it was mostly built out decades ago. The last large change was bulldozing I-93 through in the late 1960s:

[![](https://www.jefftk.com/buldozing-i93-1969.png)](https://www.jefftk.com/buldozing-i93-1969-big.png)

This makes fatality rates over time comparable in a way they wouldn't be if you were looking at a city that had expanded, contracted, or been seriously rebuilt.

I pulled accident date from the national [Fatality Analysis Reporting System](https://www.nhtsa.gov/research-data/fatality-analysis-reporting-system-fars) (FARS) which goes back to 1975. Here's what I see for the number of fatal crashes annually ([code](https://github.com/jeffkaufman/fars/blob/main/somerville-process.py)):

[![](https://www.jefftk.com/somerville-annual-fatal-crashes.png)](https://www.jefftk.com/somerville-annual-fatal-crashes-big.png)

*blue bars are annual counts; red line is a 10y centered moving average*

Over the last 40y it's gotten about 3x better, though 2019 was a pretty bad year.

I'd also be interested in whether our roads are becoming more dangerous for pedestrians specifically. Is the decrease just due to airbags, crash testing, and cars becoming safer for their occupants? While pedestrian data in FARS only goes back to 1991, we can still graph what we have:

[![](https://www.jefftk.com/somerville-annual-pedestrian-crashes.png)](https://www.jefftk.com/somerville-annual-pedestrian-crashes-big.png)

It does look like we're making progress specifically within pedestrian safety as well, though 2019 now sticks out even more.

(I'm being lazy here and charting "traffic accidents with a fatality and involving at least one pedestrian". The FARS data does make it possible to see whether it was a pedestrian that died, but this requires joining CSVs in a way that's a bit awkward. In the ones I've spot checked the fatality was a pedestrian, which is what you'd expect.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0HNMqvdXZvuwrmX4pTRjc5Jp2HxdJzbvjcd9y1rpnSiWAaxJZ8hsZoPB5m6bKCkYKl), [mastodon](https://mastodon.mit.edu/@jefftk/109915410632233894)*
# Are there rationality techniques similar to staring at the wall for 4 hours?

I'm wondering if one exists, and what is the name of that technique or that family of techniques. It seems like something that CFAR has researched extensively and layered a lot of things on top of. 

4 hours is necessary for anything substantially longer than 1-2 hours, since 6 hours is too long under most circumstances. Obviously, whiteboards and notepads are allowed, but screens and books absolutely must be kept in a different room. I'm not sure how sporadic google searching and book-searching and person-consulting factors into this, because those queries will suck you in and interrupt the state.

If people are using sleep deprivation, showers, lying in bed, or long drives to think, it's probably primarily the absence of interruption (from books and screens and people) that triggers valuable thoughts and thinking, not the tasks themselves. (although freeway driving might potentially do more good than harm by consistently keeping certain parts of the brain stimulated).
# Meta "open sources" LMs competitive with Chinchilla, PaLM, and code-davinci-002 (Paper)

As the title says, Meta trained 4 foundational models with 7B, 13B, 33B, and 65B parameters respectively, and is open sourcing them for research.[^ubd95r91ii] You can get their code on their Github repo: [https://github.com/facebookresearch/llama](https://github.com/facebookresearch/llama) but you need to fill in [a Google form](https://forms.gle/jk851eBVbX1m5TAv5) to get the weights.   
![](https://slack-imgs.com/?c=1&o1=ro&url=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FFpvTXeJXwAAGmDU.png)

On downstream benchmarks, the models do comparably well with Chinchilla and PaLM and only a bit worse than Flan-PaLM-540B and `code-davinci-002`/`text-davinci-002`. (The authors don't evaluate on those models, but you can look at their performance from other work such as Stanford's [HELM](https://crfm.stanford.edu/helm/latest/) or Chung, Hou, Longpre et al's ["Scaling Instruction-Finetuned Language Models"](https://arxiv.org/abs/2210.11416).

* * *

Abstract:

> We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B. We release all our models to the research community.

* * *

Twitter thread from authors: [https://twitter.com/GuillaumeLample/status/1629151231800115202](https://twitter.com/GuillaumeLample/status/1629151231800115202)

* * *

[Eliezer guesses](https://twitter.com/ESYudkowsky/status/1629207625933135877) that the model won't be impressive in practice:

> I blindly guess, could be wrong, that this model will turn out sufficiently unimpressive in practice that nobody uses it for much. Basically based on a guess that more than benchmarks matter, and Meta has no people competent to do the tricky stuff needed to stay on current edge.

[^ubd95r91ii]: It's not necessarily open source as you think of it -- you need to fill in a Google form, and then they might give it to you:In order to download the checkpoints and tokenizer, fill this google formThe license is intended only for non-commercial, research work:Meta grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited license under Meta’s copyright interests to reproduce, distribute, and create derivative works of the Software solely for your non-commercial research purposes.
# The Practitioner's Path 2.0: the Empiricist Archetype

Earlier this month, we [introduced](https://guildoftherose.org/articles/the-future-of-structured-self-improvement) the Practitioner's Path 2.0, our new framework for structured self improvement. The heart of the Path is the Skilltree, which is divided into three archetypes: [the Pragmatist](https://guildoftherose.org/articles/path-2-0-the-pragmatist-archetype), [the Meditative](https://guildoftherose.org/articles/path-2-0-the-meditative-archetype), and the Empiricist.

The Practitioner's Path 2.0 launches on **March 4th**. Today, we'll go over the Empiricist archetype and her associated Skills.

The Empiricists see life as a puzzle, a complex system that can be manipulated into producing a desired outcome. Empiricists love to learn new facts and skills, even if they're not immediately useful. Theorizing about the world is fun too, but what the Empiricist really wants is to be *right* — and that means checking those clever hypotheses against the only judge that really matters: reality.

In the Empiricist tree, you'll find Skills focused on:

*   Synthesizing and distilling information
*   Learning and teaching
*   Thinking clearly

Some people might call Empiricists excessively logical, but the Empiricist knows that's a mischaracterization. Intuition is just as important as reason, and the Empiricist's ultimate goal is to practice thinking clearly so much that it becomes automatic.

### Introductory Skill Examples

![](https://guildoftherosemedia.s3.us-west-2.amazonaws.com/empiricist3_ddd376c810.png?updated_at=2023-02-08T00:51:20.278Z)

![](https://guildoftherosemedia.s3.us-west-2.amazonaws.com/empiricist1_fbc52c77dc.png?updated_at=2023-02-08T00:51:20.302Z)

![](https://guildoftherosemedia.s3.us-west-2.amazonaws.com/empiricist2_8e529bcd20.png?updated_at=2023-02-08T00:51:20.300Z)

![](https://guildoftherosemedia.s3.us-west-2.amazonaws.com/empiricist4_4ef00ed8f9.png?updated_at=2023-02-08T00:51:20.208Z)

### The Archetypal Empiricist

Eva Davis started her career as a nameless chemical technician with a slowly fading passion for science. Today, ten years later, she's a max-level Empiricist advancing the field of anti-aging medicine.

At work, Eva is a brilliant lab scientist for a biotech startup, Everlife. She devises experiments, runs them, and uses the results to, someday, allow the elderly to spend more time with their children and grandchildren. The reaper's scythe-hand weakens one day at a time.

Eva spends most of her time at work, but when she allows herself a rare moment of relaxation, Eva applies her razor-sharp mind to the world around her. Knowledge is everywhere — from the science papers she reads, to the natural history museum she frequents, to the endless stream of books she devours. She stays up late into most nights reading, telling herself that the exhaustion the next day is the price of knowledge.

* * *

The Practitioner's Path 2.0 launches **March 4th**. In the meantime, what do you think of the Empiricist archetype? Does it resonate with you? Let us know in the comments below!
# Would more model evals teams be good?

If multiple model evaluations organizations similar to the ARC evals team emerge and compete for big tech clients, does this drive a "race to the bottom," where more lenient evals teams get larger market share? I'm imaging a scenario in which there are government restrictions on deployment/excess training of unevaluated models. Big tech companies would have to receive a certificate from a government-approved evaluator before training their model past a certain point, or with certain "dangerous" methods (e.g., RLHF without adequate regularizers), or deploying it off the training distribution.

In an alternative scenario, big tech companies are motivated to seek "penetration testing" from evals teams because this prevents deployment failures that tank stocks. Or, more optimistically, perhaps the big tech companies take the risk of emergent misaligned powerseeking behavior seriously and genuinely want to avoid (apparent) tail risks. In this scenario, perhaps multiple evals teams compete to be the best at finding bugs in models. However, if the most easily detectable bugs are not critical to MAPS, which could be rare and catastrophic, perhaps this competitive environment also entails a "race to the bottom" in terms of the quality of model evaluations?
# A library for safety research in conditioning on RLHF tasks

There is a couple of discussions regarding conditioning / decision transformer training. "Conditioning" as in, placing your reward model's reward in the prompt. We then train our language model to create a completion that follows the specified reward.

See [Safety considerations for online generative modeling](https://www.lesswrong.com/posts/BMfNu82iunjqKyQA9/safety-considerations-for-online-generative-modeling#Safety_advantages_of_generative_modeling), [Soft optimization makes the value target bigger](https://www.lesswrong.com/posts/9fL22eBJMtyCLvL7j/soft-optimization-makes-the-value-target-bigger#Fine_tuned_generative_models), [RLHF bad, conditioning good](https://www.lesswrong.com/posts/AXpXG9oTiucidnqPK/take-13-rlhf-bad-conditioning-good). 

The tldr is that training models this way could have safety benefits.

[I've created a library](https://github.com/thejaminator/conditionme) so that we can extend a pre-trained LLM (gpt2, gpt-j) to work with conditioning by **scalar** rewards. This allows researchers to save time by avoiding the need to modify attention masks, positions, and labels themselves. For example, researchers can retrain GPT-2 to replicate [OpenAI's summarization RLHF paper](https://openai.com/blog/learning-to-summarize-with-human-feedback/), but by relying purely on conditioning. I created it because I couldn't find an existing library that did so.

Note: an easier way of conditioning would be to use discrete tokens. [Pretraining models With Human Preferences](https://www.lesswrong.com/posts/8F4dXYriqbsom46x5/pretraining-language-models-with-human-preferences) implements conditioning via discrete 
 and 
 tokens.

However using scalar rather than discrete rewards could have the following benefits

*   The ability to specify a higher reward than the maximum reward available during training. For example, during training the maximum reward obtained was 1. However during inference you can set it to 2. Would we get a better result, as demonstrated in the [decision transformer paper](https://arxiv.org/abs/2106.01345)? (Thanks [@Tomek Korbak](https://www.lesswrong.com/users/tomek-korbak?mention=user) for pointing this out)
*   More efficient training since we don't lose information during discretization (yet to be empirically shown?)

While the library is still in early development, it can already be used for offline training of GPT-2. 

I'm writing this post earlier rather than later to:

*   Get feedback on whether this is useful. Or that it isn't useful and I should spent time on something else :)
*   Get feedback on how to improve the library if the above statement is true.
*   Save someone hours trying to do the same thing (just in case that happens)
# [Link] Petition on brain preservation: Allow global access to high-quality brain preservation as an option rapidly after death

I've put a few extracts I found notable below, but check out the URL for the full text and to sign the petition. Note that signing the petition doesn't mean you necessarily want to preserve your own brain.

Extracts I found notable:

> In 2016, the Brain Preservation Foundation Prize was awarded for a scalable technique that – for the first time – demonstrably preserved the key features of brain structure that neuroscientists know are involved in learning and memory.
> 
> \[...\]
> 
> Our best current models suggest that carefully preserving the pattern of connections within the human brain, i.e. our "connectome," alongside essential biomolecules that regulate those connections, will allow the retention and later automated reconstruction of unique and valuable information about a person’s identity, including their long-term memories.
> 
> \[...\]
> 
> Here, we are performing advocacy for brain preservationists. We call upon the global scientific, medical, and legal communities, as well as local and national governments, to immediately make changes in norms, policy, and laws in end-of-life medicine and care, to allow brain preservation to be accessible as an elective procedure at death, and for this procedure to be regulated so that it is done in a competent, sustainable, and humane way.
> 
> \[...\]
> 
> Currently, with our existing medico-legal frameworks, elective brain preservation is often delayed after death, in ways that people cannot opt out of, even after legal and medical preparations. This delay leads to unnecessary decomposition of the brain, and may irreparably damage the structures that contain our memories. These restrictive laws must be changed with haste.
> 
> \[...\]
> 
> We also understand that brain preservation can be expensive and that many of the existing methods are not currently attainable for all people at the end of life. We call upon scientists, technologists, and physicians to develop low-cost options, and for governments and philanthropists to make the necessary funds available so that all individuals who wish to pursue brain preservation at the end of life will be able to have access to it.

*PS: I provide free cryonics sign up support (and* [*a 50% discount for Alcor's 1st year of membership*](https://www.alcor.org/alcor-ice-program/)*); you can email me at* [*mathieu.roy.37@gmail.com*](mailto:mathieu.roy.37@gmail.com)*.*
# Taboo "human-level intelligence"

Maybe calls for tabooing terms can become a bit inflationary. However, talking about if and when AI will reach or surpass “human-level intelligence” can misleadingly frame the discussion, so I recommend avoiding that term.   
  
What is "human-level intelligence" and when will AI surpass it?   
  
My understanding is that there is a wide spectrum of human intelligence, and when people are discussing AI capabilities and threats, it is important to know where in that spectrum the AI in question sits.   
  
Moreover, intelligence is not a single-dimensional concept. Sure, there is *g* etc. - in human brains. But even an ordinary calculator can, well, calculate much faster than humans. For being dangerous, an AI does not need to be better than all people in every discipline. Does the AI have to be a great composer, or art historian? No.  
  
Yet the vagueness of the concept of "human-level intelligence" may lead to further misleading ways of assessing AI capabilities. Firstly, dismissing domain-specific abilities; "sure, this AI is great at explaining jokes but it cannot invent new salsa recipes, whereas this is something that humans can do." True, but the human brain is quite modular, subsystems do different things, so why should an AI not also use different AIs? Secondly, comparing with the best human experts in some domain; "look, this AI may win a game of Diplomacy, but not against the best players." Fine, but if human-level intelligence means that you have to be best at everything, then no human has human-level intelligence. This would only be relevant if humanity coordinated itself perfectly.   
  
Additionally, to be dangerous, an AI does not have to be conscious. But this is something else that the term "human-level intelligence" may suggest. Why, if the term does not contain the concept of consciousness? The reason for this is that the lack of concreteness about what human intelligence is could invite people to substitute what they believe exists in everyone's brain, regardless of their intelligence, and that is consciousness. Saying that AI transcending human-level intelligence is a relevant threshold may sound to people as if the real danger is AIs developing consciousness.  
  
So avoid the term "human-level intelligence" if you can.

*Thanks to Justis Mills for feedback on a draft of this post.*