Harley-ml/lesswrong huggingface dataset:
# Bent or Blunt Hoods?

One of the many new EV companies, [Canoo](https://www.canoo.com/), is making a van:

[![](https://www.jefftk.com/canoo-walmart.jpg)](https://www.jefftk.com/canoo-walmart-big.jpg)

Looking at the shape of it, one of the reactions [I saw](https://news.ycombinator.com/item?id=38915311) was:

> This vehicle looks like a pedestrian killing machine ... the high blunt shape of the vehicle is going to rocket pedestrians to their death in an impact.

Let's compare to a few different vans:

*   [Mercedes-Benz Sprinter](https://www.mbvans.com/en/sprinter/cargo-van) modern full-size van.
*   [Ford E-Series](https://en.wikipedia.org/wiki/Ford_E-Series) (Econoline) traditional full-size van.
*   [Toyota Sienna](https://www.toyota.com/sienna/) minivan

These are all more angled than the Canoo, but let's compare their profiles:

[![](https://www.jefftk.com/ford-mb-canoo-toyota.png)](https://www.jefftk.com/ford-mb-canoo-toyota-big.png)

All three have an approximately vertical section, followed by a bend inwards. On the Sienna that bend is at ~40", while on the other three it's ~48".

Unfortunately all of these are too high to help much. The Institute for Highway Safety [found](https://www.iihs.org/news/detail/vehicles-with-higher-more-vertical-front-ends-pose-greater-risk-to-pedestrians) that for hoods 40"+, a sloped vs blunt profile doesn't matter: they're both ~45% more dangerous than a traditional car's low and sloped front. Which makes sense intuitively: the bend would need to be below your center of gravity, and for that to be above 40" you generally need to be >6ft.

One thing this design does have going for it [relative](https://twitter.com/FreckleEars/status/1624137853872574475) to the many new vehicles with tall blunt front ends is good forward visibility. Because there's no engine or frunk they can make the bottom front panel transparent:

[![](https://www.jefftk.com/canoo-front-window.jpg)](https://www.jefftk.com/canoo-front-window-big.jpg)

While I'm not enthusiastic about the general shift towards taller and blunter front ends, delivery vans do seem like the sort of place where they make the most sense. A long sloping front means you either need a longer van or a lower capacity, which is why old vans like the original [VW Bus](https://en.wikipedia.org/wiki/Volkswagen_Type_2#T1) or [Ford Taunus Transit](https://en.wikipedia.org/wiki/Ford_Transit#Taunus_Transit_(1953)) were flat-fronted.

[![](https://www.jefftk.com/vw-bus-type-2.jpg)](https://www.jefftk.com/vw-bus-type-2-big.jpg)

[![](https://www.jefftk.com/original-ford-transit.jpg)](https://www.jefftk.com/original-ford-transit-big.jpg)
# The Perceptron Controversy

Connectionism died in the 60s from technical limits to scaling, then resurrected in the 80s after backprop allowed scaling. The Minsky–Papert anti-scaling hypothesis explained, psychoanalyzed, and buried.

I wrote it as if it's a companion post to Gwern's [The Scaling Hypothesis](https://gwern.net/scaling).
# Prediction markets are consistently underconfident. Why?

look at this calibration s-curve from [calibration.city](https://calibration.city)

![](https://www.notion.so/image/https%3A%2F%2Fprod-files-secure.s3.us-west-2.amazonaws.com%2F929e41de-47b1-4e36-bda9-1cb5af530b85%2F5c37588e-5ff1-44af-83c8-0026466f83b3%2FUntitled.png?table=block&id=42996690-3e23-4cee-ae71-aaa576009351&spaceId=929e41de-47b1-4e36-bda9-1cb5af530b85&width=2000&userId=3e94007a-f5e8-443b-9a5d-376498ae5983&cache=v2)

markets are consistently closer to 50% than should be. why?

I have my own guesses but curious to hear yours.
# Introduce a Speed Maximum

Speeding is one of the most common ways for Americans to break the law. Drive the speed limit on the highway around here and you'll typically be the slowest car on the road. How much over the speed limit is customary varies regionally, but drivers often expect cops to ignore them at 5-15 mph over.

Overall, I think this is a pretty bad situation. It gets people used to ignoring laws, people who scrupulously follow the law are often at higher risk (and cause higher risk to those around them) than if they went along with traffic, driverless cars go awkwardly slow, some risk of selective enforcement, confusing for travelers, etc. How can we get out of this?

If we just started strictly enforcing the current limits we'd have a mess: it's too big a behavior change to push all at once so you'd see even more dangerous variance in speeds than today, and it's unclear we actually want people driving the posted speeds. It also wouldn't work well to raise the limit to the speed people are mostly going, since many people would assume they can then go an extra 5-15mph on top of that.

Instead we could take inspiration from [Brazil](https://www.npr.org/sections/money/2010/10/04/130329523/how-fake-money-saved-brazil) and introduce a parallel system of maximum speeds:

[![](https://www.jefftk.com/speed-limit-55-max-65.png)](https://www.jefftk.com/speed-limit-55-max-65-big.png)

Initially this has no legal effect, and just makes the existing amount of leeway more legible. On a 55mph road where people normally drive 60-65 and the police don't start ticketing until you're more than 10mph over, the signs would say both "speed limit 55" and "max 65". These would be rolled out gradually, in consultation with traffic engineers and the people responsible for enforcement.

As they roll out, you adjust enforcement to match. Put up speed cameras set to the maximum in many places, and in other places have police enforce the max strictly after each sign is put up. Traveling above the limit but below the maximum becomes effectively allowed, since there's no enforcement.

Once the rollout is complete you overhaul the laws around speeding to make the maximum the legal limit, and adjust rules that are set relative to the old limit to still make sense. For example, if you previously gave only low fines for going 58 in a 55 zone, and in practice never issued them, while you gave high fines for going 68, you would still want the higher fine for going 68 in a "max 65" zone. The goal is to bring the law in line with behavior, but otherwise keep the status quo.

At this point you could consider removing the older lower "speed limit" signs, but I think it's probably worth keeping them as advice about what speed to travel. In some cases you might raise them a bit, knowing that with the maximum in place as a firm limit you'll get slightly faster speeds but lower variance.

I think there's a path here that brings the law back in line with driver and enforcement behavior, while otherwise essentially maintaining the status quo. It does require new signs and some policy tweaks, but seems on balance pretty positive to me.
# Decent plan prize announcement (1 paragraph, $1k)

Edit Jan 20: [Winner & highlights](https://www.lesswrong.com/posts/fEuZdHbCofHKgjmCQ/decent-plan-prize-winner-and-highlights)

Say I'm about to do a real big training run on playing video games, predicting text, predicting physics, writing code that works, etc etc. Say I've got a real good neural net architecture and a whole lot of flops. Say I'm a company and I'm gonna use this thing for AI lawyers and coders etc for a profit. Say I'm mildly concerned it somehow kills me and am willing to throw a few $ to prevent that.

So what should I do? How should I train and deploy the model?

Comment below or answer [at this link](https://docs.google.com/forms/d/17SS5_iPppeGYnKiOlSpTstRNFutyZCQ8xkh-hfgsss4/viewform) if you don't want to be plagiarized.

Prize goes to best answer. (I judge obviously.)

The shorter the answer the better.

Deadline is Wednesday January 17 anywhere on Earth but answering immediately is better/easier.

You may accept your prize as 50 pounds of quarters if you prefer.

* * *

Clarification jan 12: say I've got 1000x the gpt4 flops and that my architecture is to transformers as convolutions are to simple MLPs in vision (ie a lot better)

Clarification 2:  an answer like "here's how to get strong evidence of danger so you know when to stop training" is valid but "here's how to wipe out the danger" is much better.

3: Example answer for nuclear generators: "Spring-load your control rods so they are inserted when power goes out. Build giant walls around reactor so if steam explodes then uranium doesn't go everywhere. Actually, use something low pressure instead if possible, like molten salt or boiling water. Lift the whole thing off the ground to avoid flood risk."

4: This is hypothetical. I am not actually doing this. I'm not a billionaire.

5: "Hire someone" and "contract it out " and "develop expertise" etc obviously do not count as answers.
# What good is G-factor if you're dumped in the woods? A field report from a camp counselor.

I had a surprising experience with a 10 year old child "Carl" a few years back. He had all the stereotypical signals of a gifted kid that can be drilled into anyone by a dedicated parent- 1500 chess elo, constantly pestered me about the research I did during the semester, used big words, etc. This was pretty common at the camp. However, he just felt different to talk to- felt sharp. He made a serious but failed effort to acquire my linear algebra knowledge in the week and a half he was there.   
  
Anyways, we were out in the woods, a relatively new environment for him. Within an hour of arriving, he saw other kids fishing, and decided he wanted to fish too. Instead of discussing this desire with anyone or acquiring a rod, he crouched down at the edge of the pond and just watched the fishes. He noticed one with only one eye, approached it from the side with no vision, grabbed it, and proudly presented it to the counselor in charge of fishing.   
  
Until this incident I was basically sceptical that you could dump some Artemis-Fowl-figure into a new environment and watch them big-brain their way into solving arbitrary problems. Now I'm not sure. 

His out-of the box problem solving rapidly shifted from winning camper-fish conflicts to winning camper-camper conflicts, and he became uncontrollable. I almost won by breaking down the claim "You have to do what I say" into "You want to stay at camp, here's the conditions where that happens, map it out- you can see that you're close to the limit of rules broken where you still get what you want." This bought two more days of control. Unfortunately, he seems to have interpreted this new system as "win untracably," and then was traced trying to poison another camper by exploiting their allergy. He's one of two campers out of several thousand I worked with that we had to send home early for behavior issues. 

In the end, he was much less happy than the other campers I've had, but I also think he's one of the few that could survive "Hatchet" or "Call of the Wild" style- despite comparative lack of experience.  
  
Addendum: he harassed and kept catching the poor half-blind fish for the duration of the stay, likely because he got so much positive attention the first time he caught it.
# Concrete examples of doing agentic things?

I've been thinking about agency recently. **I want to be more agentic!**

Sometime last year, I read [Cultivating And Destroying Agency](https://www.lesswrong.com/posts/rXX6vd2RrKHfePPfE/cultivating-and-destroying-agency) and [Seven ways to become unstoppably agentic](https://www.lesswrong.com/posts/Tnpp3cyEHMGthjGAf/seven-ways-to-become-unstoppably-agentic). I recently saw [How to be More Agentic](https://usefulfictions.substack.com/p/how-to-be-more-agentic) and it kind of spurred me to write this post (which is actually a form of agency). I credit these posts with expanding what I think is possible. What I found really helpful was the concrete examples of the things they did (you can just buy stuff online, you can cold email people, try to get as many rejections as possible, sign up to take a test w/o taking a class).

These examples felt really helpful because they expanded the domain of things I thought were possible. I want to expand my agency Overton window.

My model is that there are two parts to expanding agency:

1.  **Find out** or **think of** things that you ***can*** **do** that you had not thought of before (because they are uncommon / not what society tells us to do / require [some staring into the abyss](https://www.lesswrong.com/posts/vzfz4AS6wbooaTeQk/staring-into-the-abyss-as-a-core-life-skill))
2.  **Actually do them**, get good results, and have a positive feedback loop

If you have examples of things that you have done are just totally orthogonal to what we think of normal, that were ultimately helpful (or not), please put them here. This will hopefully help with step 1. Actually Doing Things and getting a positive feedback loop requires effort (but it hopefully is fun and rewarding).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fc9a53765ae9218a7cd86ec195e0196a605afffa228e75f9.png)

If what you did ended really badly, please still put it, but just also put that it ended badly. I can reverse advice. If it ended well, say that also! The more examples the better. My main goal is to **expand my** ***possibility hull*** (pictured above). If you are unsure whether the thing you are saying is already known, [please say it anyways](https://www.lesswrong.com/posts/vYzLuRndnMACHM9HZ/obvious-advice).
# Why do so many think deception in AI is important?

I see a lot of energy and interest being devoted toward detecting deception in AIs, trying to make AIs less deceptive, making AIs honest, etc. But I keep trying to figure out why so many think this is very important. For less-than-human intelligence, deceptive tactics will likely be caught by smarter humans (when a 5-year-old tries to lie to you, it's just sort of sad or even cute, not alarming). If an AI has greater-than-human intelligence, deception seems to be just one avenue of goal-seeking, and not even a very lucrative or efficient one.

Take the now overused humans-to-chimpanzee analogy. If humans want to bulldoze a jungle that has chimpanzees in it, they will just bulldoze the forrest, and kill or sell any chimps that get in their way. They don't say "okay, we're going to take these sticks of dynamite, and conceal them in these bundles of bananas, then we'll give the bananas to the chimps to earn their trust, and then, when the time is right, we'll detonate them." You just bulldoze the forrest and kill the chimps. Anything else is just needlessly convoluted.[^urgrtmkr2a]

If an AI is smart-enough to deceive humans, and it wants to gain access to the grid, I don't see why it wouldn't just hack into the grid. Or the internet. Or server farms. Or whatever it's trying to get.

What am I missing? What situation in the future would make detecting deception in models important?

[^urgrtmkr2a]: Ironically, deceptive tactics in this case would likely correlate with niceness. If you want to peacefully relocate the chimps without disturbing or scaring them, then you might use deception and manipulation. But only if you actually care about their wellbeing.
# Takeaways from the NeurIPS 2023 Trojan Detection Competition

This link summarizes our [research takeaways](https://confirmlabs.org/posts/TDC2023) as participants (and winners in one of the four tracks) of the NeurIPS 2023 Trojan Detection Competition, a competition about red-teaming LLMs and reverse-engineering planted exploits.
# Vote With Your Face

Starting with our July 2nd dance [BIDA](https://www.bidadance.org/) has been running half our dances as [mask-optional](https://blog.bidadance.org/2023/06/some-mask-optional-dances.html). Some people only want to attend if they know everyone on the floor will be wearing a [high-filtration mask](https://blog.bidadance.org/2022/05/requiring-high-filtration-masks.html), others only want to attend if they don't have to mask, and of course others will or won't attend regardless. Offering some of each lets people choose the dances that fit their preferences.

Plus, as someone who likes experiments, it's almost like a randomized controlled trial of mask requirements on attendance!

Before we switched to alternating masks required and optional we didn't know what effect this would have on attendance: would there be more people on mask-optional nights (more fun!) or on mask-required nights (more safe!) Here's what I see looking back over attendance:

[![](https://www.jefftk.com/bida-attendance-by-mask-status.png)](https://www.jefftk.com/bida-attendance-by-mask-status-big.png)

Note that aside from there only being eleven dances, there are some unique aspects of some of the dances that you might want to adjust for:

*   The 7/2 dance was the first mask-optional dance, and so might have gotten a boost from people who were excited to dance without masks but wouldn't come every time.
    
*   The 7/2 and 9/3 dances were our last dances before the summer and first dance back, which could have increased attendance. But they were also both holiday weekends, which could have decreased attendance.
    
*   The 11/5 and 11/19 dances were both double dances, which generally have somewhat higher attendance. But we (mostly by accident) did two back to back, one of each.
    
*   The 12/3 dance was a family dance, which also usually means higher attendance.
    
*   The 12/17 dance was at a different venue, the Somerville Armory, which usually means lower attendance.
    
*   The 1/7 dance was during a major snowstorm, which usually means much lower attendance.
    

Overall, it looks like mask-optional nights are a bit more popular, but with all the caveats it's hard to say. Regardless, the difference clearly not so large that it's a reason to stop one or the other. I expect we'll continue alternating, since we still hear from many people that they only want to attend one or the other.

(All this is speaking for myself, not the BIDA Board.)
# What does it look like for AI to significantly improve human coordination, before superintelligence? 

I'm currently thinking through the tech tree of developments in human coordination that would increase the probability that the development of systems that are superintelligent (in some to all domains), goes well. 

Just to start off the brainstorming, here are some:

*   AI in negotiation opens many interesting avenues, such as: 
    *   Allowing parties to reveal confidential or sensitive information to a common, trusted AI, that they normally couldn't reveal to either a negotiation partner or a mediator
    *   Having an AI search the space of compromises or "third options" way faster and deeper than humans could
    *   Allowing AIs to build a model of your preferences over time, in such a way that two AIs could instantly "pre-negotiate" and suggest a solution, for issues that otherwise wouldn't be worth investing the effort into
*   ["ML clean rooms"](https://aws.amazon.com/clean-rooms/ml/) might allow actors to run evaluations on each others models, without having to reveal weights or other sensitive data. 
*   Monitoring technology (such as by massive analysis of comprehensive satellite data) might allow novel agreements that weren't previously enforceable
*   Human lie detection tech (note: highly dual-use)

It is an open question which, if any, coordination tech we will be able to build with the level of AI that will be available at different times in the future, and which of it will be available before its [too late to be useful](https://www.lesswrong.com/posts/n3w3ww9Xuf8SngBfE/replacement-for-ponr-concept#comments). 

It's also an open question whether there's any specific tech that will be more helpful than just the normal bitter lesson of "GPT-n+1 doing with a good prompt what GPT-n did with all kinds of bells and whistles". 

Nonetheless, whether it be nation state level agreements, inter-lab agreements, or just teams within a lab deciding what to work on: the potential upside from improved human coordination is pretty massive. 

So, I'm curious for people's thoughts: 

**What are potential ways AI could be used to significantly improve human coordination, between now and the leadup to superintelligence?  **

(Sidenote 1: I'll also accept answers for non-AI coordination tech that would still serve as an important enabler here, such as "something something better commitment mechanisms")

(Sidenote 2: My interest in this is more from the angle of a potential founder, than a researcher: I want to know if there are any potentially extremely promising technologies that I or someone else should try building)
# Introducing REBUS: A Robust Evaluation Benchmark of Understanding Symbols

*This is a summary of* [*https://arxiv.org/abs/2401.05604*](https://arxiv.org/abs/2401.05604)*.*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b3f5b57fa2d51289e7e158aff4a64c3755fb49ca45b6ea40.png)

When Google announced Gemini Pro, they [displayed its ability to solve rebuses](https://www.youtube.com/watch?v=aRyuMNwn02w)—wordplay puzzles which involve creatively adding and subtracting letters from words derived from text and images.

We introduce a new benchmark ([Github](https://github.com/cvndsh/rebus)) evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories like movies, composers, major cities, and food. 

The REBUS dataset highlights several key challenges in multimodal language models:

*   **Multi-step visual reasoning—**many rebuses contain information in a meaningful pattern, from which the necessary string operations and structure must be successfully inferred.
*   **Spelling—**string manipulations require accurate letter-wise representations.
*   **Hypothesis testing**—for instance, if the model recognizes a fictional-character-themed puzzle as containing images representing “Megachiroptera” and “Einstein,” it needs to revise its initial interpretations to reach the correct answer “Batman.”
*   **World knowledge—**many puzzles contain crucial references and hints.
*   **Grounded image recognition—**puzzles sometimes require identifying the most important part of an image, or recognizing what the image may represent as a whole, like understanding that a photograph of a group of lions might be cluing “lion,” “big cats,” “savanna,” or “pride.”
*   **Understanding human intent—**solving the puzzles requires an understanding of what answers or reasoning steps could have been plausibly developed by the puzzle author.

We find that the proprietary models GPT-4V and Gemini Pro significantly outperform all other tested models, but even they only achieve scores of 24% and 13.2%, respectively. Models rarely understand all parts of a puzzle, and they almost always fail to explain their correct answers with correct justifications.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4cc6498687c0f1b6d6ec0d818bb20ff4d6ef68bfb1253106.png)
# Why are people unkeen to immortality that would come from technological advancements and/or AI?

Nearly everyone with whom I have talked about the mindblowing possibilities of a friendly ASI -people that hadn't heard about it before-, I have seen the same reaction: the person is skeptical and rejects the idea of immortality: "Humans are made to die", "what is the value of life then?", and sometime: "I want to die".

**Why does that happen?** I can't understand their reaction.

If you are like them, let us know your arguments.
# AlphaGeometry: An Olympiad-level AI system for geometry

\[Published today by DeepMind\]

> **Our AI system surpasses the state-of-the-art approach for geometry problems, advancing AI reasoning in mathematics**
> 
> Reflecting the Olympic spirit of ancient Greece, [the International Mathematical Olympiad](https://www.imo-official.org/) is a modern-day arena for the world's brightest high-school mathematicians. The competition not only showcases young talent, but has emerged as a testing ground for advanced AI systems in math and reasoning.
> 
> In a paper published today in [*Nature*](https://doi.org/10.1038/s41586-023-06747-5), we introduce AlphaGeometry, an AI system that solves complex geometry problems at a level approaching a human Olympiad gold-medalist - a breakthrough in AI performance. In a benchmarking test of 30 Olympiad geometry problems, AlphaGeometry solved 25 within the standard Olympiad time limit. For comparison, the previous state-of-the-art system solved 10 of these geometry problems, and the average human gold medalist solved 25.9 problems.

Links to the paper appear broken, but here is a link:

[https://www.nature.com/articles/s41586-023-06747-5](https://www.nature.com/articles/s41586-023-06747-5)

Interesting that the transformer used is tiny. From the paper:

> We use the Meliad library for transformer training with its base settings. The transformer has 12 layers, embedding dimension of 1,024, eight heads of attention and an inter-attention dense layer of dimension 4,096 with ReLU activation. Overall, the transformer has 151 million parameters, excluding embedding layers at its input and output heads. Our customized tokenizer is trained with ‘word’ mode using SentencePiece and has a vocabulary size of 757. We limit the maximum context length to 1,024 tokens and use T5-style relative position embedding. Sequence packing is also used because more than 90% of our sequences are under 200 in length.
# More Usable Recipes

Instead of a recipe book I keep a [web page](https://www.jefftk.com/recipes/). Whenever I make something and think I'll make it again, I add it. It's public since I generally make things public by default, though it's terse since it's intended as a reference for someone (me!) who already knows what to make. Recipes looked something like:

[![](https://www.jefftk.com/recipes-traditional.png)](https://www.jefftk.com/recipes-traditional-big.png)

I've now updated them so I can check things off as I go:

[![](https://www.jefftk.com/recipes-usable.png)](https://www.jefftk.com/recipes-usable-big.png)

Since I [author in HTML](https://www.jefftk.com/p/hand-writing-mathml), I didn't want to turn my tidy source from:

<li>2 eggs (or 2T flax and 5T water)
<li>2/3 oil
<li>1C greek yoghurt
<li>1/4 cup milk, more if needed

Into something ugly like:

<li><label><input type=checkbox></input>
    2 eggs (or 2T flax and 5T water)</label>
<li><label><input type=checkbox></input>
    2/3 oil</label>
<li><label><input type=checkbox></input>
    1C greek yoghurt</label>
<li><label><input type=checkbox></input>
    1/4 cup milk, more if needed</label>

Instead, I've added a little JS to each page that does this at display time:

for (const li of
     document.getElementsByTagName("li")) {
  li.innerHTML =
     "<label><input type=checkbox></input>" +
     li.innerText + "</label>";
  }
}

I've also added a bit of CSS for extra vertical space for the checkbox but only on devices like phones where big sloppy fingers need it:

<style>
@media (pointer:coarse) {
  input {
    margin-top: 1em;
    margin-bottom: 1em;
  }
}
</style>

I haven't actually cooked anything since changing this, but I think I'll like it.
# Against Nonlinear (Thing Of Things)

[Against Nonlinear](https://thingofthings.substack.com/p/against-nonlinear) is a followup that Ozy wrote to [Practically A Book Review: Appendix to "Nonlinear's Evidence: Debunking False and Misleading Claims"](https://thingofthings.substack.com/p/practically-a-book-review-appendix). In it, they lay out 4 trends in Effective Altruism where Nonlinear was especially bad:

*   Do It For The Gram: Nonlinear was turning Effective Altruism into being Instagrammable instead of making the world better
*   So-Called Top EAs: Nonlinear was ranking EAs based on their participation in exclusive parties
*   File Your Paperwork: Nonlinear was disregarding legally mandated paperwork or generally important accounting
*   Sunshine Is The Best Disinfectant: Nonlinear was exploiting and undermining EA's trustworthiness
# Are there high-quality surveys available detailing the rates of polyamory among Americans age 18-45 in metropolitan areas in the United States?

In recent years, there has been more public attention paid to seemingly increasing rates of less common identity characteristics or alternative lifestyle choices. This has provoked widespread media speculation that the trends in question are social contagions being spread by the communities or subcultures where such trends are most apparent. Examples of such trends include dramatically increased rates of reported self-identification with sexual or gender minority status; various kinds of neurodivergence, and polyamory/non-monogamy. Without denying the possibility that 'social contagion' may be a minor causal factor among many for increased rates of self-identification with the above characteristics, there are of course better explanations. Yet the narrative that these trends are becoming more prominent mostly as a consequence of social pathology persists. This specious presumption could be significantly mitigated by evaluating general causes, which in turn might be identified through knowing more about the base rates, and increase in the rate of change, for these personal/social characteristics among the general population.

Of course, on LessWrong, this will bring to mind how higher rates of polyamory among the rationality community, and related communities like effective altruism, are exploited to stigmatize them by various hostile parties. Yet I've noticed this is an even greater issue faced by larger social classes among younger generations. While public discourse has also provoked some defence of various subcultures embracing polyamory, I've noticed almost nobody on any side of the conversation bothers evaluating whether there's a general increase in rates of non-monogamy uncorrelated with increased rates of membership in alternative subcultures. To have such information would be useful for answering many unresolved questions about the status of non-monogamy in society at large.

I request data on a population sample with the particular characteristics mentioned in the title because those are the common characteristics of the non-monogamous people generally subject to so much public speculation. I'm not super well-versed in the finer details for what counts as a "higher-quality" survey, so I welcome you to make that judgment for yourself. Suggestions for other online spaces where I could ask this question which may have more insight are also welcome.
# What rationality failure modes are there?

How do people fail to improve their rationality? How do they accidentally harm themselves in the process? I'm thinking of writing a post "How not to improve your rationality" or "A nuanced guide to reading the sequences" that preempts common mistakes, and I'd appreciate hearing people's experiences. Some examples:

- It took me an absurdly long time (like, 1-2yr in the rat community) before I realized you don't correct for cognitive biases, you have to "be introspectively aware of the bias occuring, and remain unmoved by it" (as Eliezer put it in a podcast)
- More generally, people can read about a bias and resolve to "do better" without concretely deciding what to do differently. This typically makes things worse, e.g. I have a friend who tried really hard to avoid the typical mind fallacy, and accidentally turned off her empathy in the process.
- The implicit frame rationalists push is logical and legible, and can lead to people distrusting their emotions. And I think it's really important to listen to listen to ick feelings when changing your thought processes, as there can be non obvious effects.
  - E.g. My friend started thinking about integrity in terms of FDT, and this disconnected it from their motivational circuits and they made some pretty big mistakes because of it. If they'd listened to their feeling of "this is a weird way to think" this wouldn't have happened. 
  - (I think many people misinterpret sequence posts and decide to change their thinking in bad ways, and listening to your feelings can be a nice emergency check.)
# [linkpost] Self-Rewarding Language Models

> Abstract: We posit that to achieve superhuman agents, future models require super- human feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study, this work opens the door to the possibility of models that can continually improve in both axes

This seems like a significant step towards recursive self-improvement. Not directly changing the optimization algorithm or the weights, but this is a proxy for that. Now, if the network had some deceptive behavior, it seems (from what I understand) that it could become more deceptive by providing itself training data that will cause it to become more deceptive. I haven't thought about it much, but it seems this could also cause some weird feedback loops where flukes (or random fluctuation) in the grading could cause the models to go in some weird directions since the grading model is the n-1th model:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3da3d5996539154003e6492d81b9f789b3a5b535683bdca3.png)
# Why have insurance markets succeeded where prediction markets have not?

Insurance is big business and is load-bearing for many industries. It has gained popular acceptance. Prediction markets have not. This is despite clear similarities between the two domains.

One can list similarities:

*   *You are trading financial contracts about the odds of real world events.*
*   The market price of a policy elicits and aggregates information about the forecasts of professional traders and this price can inform the publics' view of risk.
*   Insurance is a tax on BS (If one underwriter says to you "that policy you wrote, its a steal, that boat's never gonna sink" you can say "wanna bet?" and let them reinsure you.)

Or you can consider this extract from an official history of Lloyd's of London[^7d0d3ej0arv]:

> It was possible to get a policy - which is a dignified way of saying a bet - on almost anything. You could get a policy on whether there would be a war with France or Spain, whether John Wilkes would be arrested or die in jail, or whether some Parliamentary candidate would be elected. Underwriters offered premiums of 25 per cent on George II's safe return from Dettingen; there were policies on whether this or that mistress of Louis XV would continue in favour or not.
> 
> One particularly grisly form of speculation, quoted by Thomas Mortimer in a book published in 1781 and engagingly entitled The Mystery and Iniquity of Stock Jobbing, was this:
> 
> \> A practice likewise prevailed of insuring the lives of well-known personages as soon as a paragraph appeared in the newspapers, announcing them to be dangerously ill. The insurance rose in proportion as intelligence could be procured from the servants, or from any of the faculty attending, that the patient was in grave danger. This inhuman sport affected the minds of men depressed by long sickness; for when such persons, casting an eye over a newspaper for amusement, saw that their lives had been insured in the Alley at 90 per cent, they despaired of all hopes; and thus their dissolution was hastened.
> 
> It is possible that at Lloyd's such activities where limited to a few. Even so the reputation of the coffee house was endangered.

***So, given the many and obvious similarities between insurance markets and prediction markets, why have insurance markets succeeded where prediction markets have not?***

[^7d0d3ej0arv]: Taken from pages 26-28 of Hazard Unlimited by Anthony Brown.
# Who Organizes Dances?

Most contra dances are organized by a committee of volunteers: ~3 to ~8 people who divide up the work. This includes both advance work like coordinating with the venue, booking performers, and publicizing the dance, and also day-of work like setup, cleanup, and handling things that come up at the dance. For a detailed example, you could look at BIDA's [current division](https://bidadance.org/board-roles).

This is not the only way to organize dances: some events are put on by a house band, house caller, or an individual (volunteer or professional). I have the impression this used to be more common before the 1970s-on revival, but don't have anything hard to go on.

Someone wrote me to ask about the typical composition of organizing committees: are they usually made up of callers? Musicians? Dancers? People who do [everything](https://www.jefftk.com/p/learning-every-part)? I decided to look back over BIDA's historical board, and see what it's been in our case.

Over the past fifteen \[1\] years we've had 34 board members. Looking over the list I remember all of them dancing at least some, and most of them also called or played music. Let's make some tables, considering only what people did during the time they were on the board: \[2\]

| non-musician | 16 (47%) |
| --- | --- |
| casual musician | 5 (15%) |
| booked to play | 13 (38%) |

| non-caller | 23 (68%) |
| --- | --- |
| casual caller | 2 (6%) |
| booked to call | 9 (26%) |

Overall, it looks like we've has a lot more musicians than callers. But do the musicians and callers tend to be the same people? Let's cross these tables:

|     | non-caller | casual caller | booked to call |
| --- | --- | --- | --- |
| non-musician | 10  | 1   | 5   |
| casual musician | 2   | 0   | 3   |
| booked to play | 11  | 1   | 1   |

It looks like the single most common role is a dance musician who doesn't call, at 11 (32%), followed closely by people who don't do either, at 10 (29%).

Losing a bit of resolution, let's categorize everyone by their primary skill-set. For example, even though I like dancing and was booked to call, my main non-organizer role was as a musician:

[![](https://www.jefftk.com/bida-board-composition-2009-2024.png)](https://www.jefftk.com/bida-board-composition-2009-2024-big.png)

| Musician | 13 (38%) |
| --- | --- |
| Dancer | 12 (35%) |
| Caller | 8 (24%) |
| Sound | 1 (3%) |

I'd be curious what this looks like in other communities!

  
\[1\] Really! BIDA started in late 2008, and our first dance was in February 2009. We're holding a big celebration dance on 2024-02-04, and if you've ever played for BIDA you're welcome to come back and join the band!

\[2\] I'm counting someone as "casual" if they do this form of some but aren't to where they'd be booked for a dance. Marking someone as a "non-musician" only means they don't play contra music—they might play an unrelated genre.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02D7d5PJLoGNFzc6MB9AHdcnPx53yJaKPsT7U9Gw9LaWkEkMpVTAqLbBkUHnpxVLNgl), [mastodon](https://mastodon.mit.edu/@jefftk/111800058821826387)*
# What environmental cues had you not seen them would have ended in disaster?

I used to think I lacked situational awareness. A search for cures returned no direct method to improve it. This is because we can perceive the same situation, but differently. When we're interested in something (like the model of a friend's new car), we find it more often. It's pulled out of our visual input instead of dying off during denoising – like in [frequency illusions](https://en.wikipedia.org/wiki/Frequency_illusion). 

To a large extent, attuning ourselves to more important things is "situational awareness". There may be other factors, like our basal arousal or how drunk we are, but variation in those seems to zero out in comparative daily importance.  

Since it's about attention, we can orient our own eyes to see things differently – creating selective frequency illusions. Learning a new word can make it appear more often in media, since the sound has sudden meaning, but the effect can also be used as a form of proactive prevention. The more complex your environment, the more likely you don’t notice a sign before too late.  

What signs have you learned to look for?

To offer some slight reframings to the original question and more specific categories: 

*   What are the most useful general orientations for broader situational awareness? 
    *   What do you want every child to know?
    *   If it meant you would otherwise forget, what words would you pin to your bathroom mirror?
*   What subtle patterns in other people spell nothing good?  
    *   What hints of recklessness, Machiavellianism, predatory tendencies, or destructive instability are subdued by politeness?
    *   Some traits of this variety are heavily offset by charm or charisma. How do you tell the difference?
*   What have you noticed before someone got hurt? 
    *   If it was a medical emergency:
        *   Vocal cues?
        *   Behavioral tics?
        *   An environmental cue they didn't notice, perhaps repeatedly?
        *   Are there very early warning signs of some conditions?
    *   If it was due to someone else, how would you warn them without making them more reticent?
*   On the road, what do you see that others miss?
    *   Aside from the danger of other drivers, what vitally important environmental cues are specific to driving?
*   What do you consider basic common sense to not ignore?
# Global LessWrong/AC10 Meetup on VRChat

This Saturday at 12:00 PM PST, we're hosting a meetup in VRChat. VRChat can run on most gaming PCs and some Android phones - and VR equipment is NOT required to use it. 

The meetup will be held in this VRChat world: [http://tinyurl.com/4df87mt4](http://tinyurl.com/4df87mt4)

If you would like a notification when it starts, please join the Virtually Rational discord ([https://discord.gg/jkD2mwP9](https://discord.gg/jkD2mwP9)).
# This might be the last AI Safety Camp

*Update Dec 4:  *Funds [still needed](https://manifund.org//projects/10th-edition-of-ai-safety-camp?tab=comments#ef1e7429-218f-4925-a5cd-89c4a01a9de6) for next month's stipends, plus salaries to run the 11th edition. Also, Zvi [listed AISC](https://thezvi.substack.com/p/the-big-nonprofits-post?open=false#%C2%A7ai-safety-camp) at the top of his recommendations for talent funnel orgs. 

* * *

We are organising the 9th edition without funds. We have no personal runway left to do this again. We will not run the 10th edition without funding. 

In a nutshell:

1.  Last month, we put out AI Safety Camp’s [funding case](https://www.lesswrong.com/posts/ukCFkqiDFmkYTyywN/funding-case-ai-safety-camp-1).   
    A private donor then decided to donate €5K.   
     
2.  Five more donors offered $7K on [Manifund](https://manifund.org/projects/10th-edition-of-ai-safety-camp?tab=bids).   
    For that $7K to not be wiped out and returned, another **$21K in funding is needed**. At that level, we may be able to run a minimal version of AI Safety Camp next year, where we get research leads started in the first 2.5 months, and leave the rest to them.  
     
3.  The current edition is off to a productive start!   
    A total of 130 participants joined, spread over 26 projects. The [projects](https://aisafety.camp/#Projects) are diverse – from agent foundations, to mechanistic interpretability, to copyright litigation.  
     
4.  Our personal runways are running out.   
    If we do not get the funding, we have to move on. It’s hard to start a program again once organisers move on, so this likely means the end of AI Safety Camp.  
     
5.  We commissioned Arb Research to do an [impact assessment](https://forum.effectivealtruism.org/posts/CuPnmeS4v5sFE6nQj/impact-assessment-of-ai-safety-camp-arb-research).   
    One preliminary result is that AISC creates one new AI safety researcher per around $12k-$30k USD of funding. 

How can you support us:

*   **Spread the word**. When we tell people AISC doesn't have any money, most people are surprised. If more people knew of our situation, we believe we would get the donations we need.
*   **Donate**. Make a donation through [Manifund](https://manifund.org/projects/10th-edition-of-ai-safety-camp?tab=comments) to help us reach the $28K threshold.  
    Reach out to remmelt@aisafety.camp for other donation options.
# Will quantum randomness affect the 2028 election?

This came up as a tangent from [@habryka](https://www.lesswrong.com/users/habryka4?mention=user) and me discussing whether [The Hidden Complexity of Wishes](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/) was correct.

**Is the result of a US presidential election 4 years from now >0.1% contingent on quantum randomness (i.e. is an otherwise omniscient observer forecasting the 2028 election today capable of >99.9% confidence, or is there >0.1% irreducible uncertainty due to quantum mechanics observer-effects)?**

I think the answer is yes, because chaotic systems will quickly amplify this randomness to change many facts about the world on election day.

*   Quantum randomness causes different radioactive decays, which slightly perturb positions of particles around the world by a few nanometers.
*   Chaotic systems will quickly amplify these tiny perturbations into macro-scale perturbations:
    *   Weather doubles perturbations every 4 days or so
    *   The genes of ~all babies less than 3 years old will be different
*   Many events relevant to the election are contingent on these differences
    *   Weather-related natural disasters, other circumstances like pandemics (either mutation or lab leak), political gaffes by candidates, assassinations (historically >0.1% and seem pretty random), cancer deaths, etc.
*   If even a small proportion of election variance is random, you get more than 0.1% election randomness.
    *   Say humanity's best estimates for the vote margin of the 2028 election have a standard deviation of 76 electoral votes centered on 0. Even if 90% of variance is in theory predictable and only 10% is true randomness (aleatoric), then the nonrandom factors have s.d. 70 and random factors have s.d. 24. If nonrandom factors have 1sd influence, random factors will flip the election with probability [well over 0.1%](https://www.wolframalpha.com/input?i=p%28x+%3E+70%29+where+x+%7E+normal%280%2C+24%29).
    *   In reality it's much worse than this, because we haven't even identified 2 leading candidates.

Oliver thinks the answer is no, because in a system as large and complicated as the world, there should be some macro-scale patterns that survive, and an omniscient observer will pick up on all such patterns. Humans are limited to obvious patterns like economic trends and extrapolating polls, but there are likely way more patterns than this, which a forecaster could use to accurately get over well 99.9% confidence.

Who is right?