Harley-ml/lesswrong huggingface dataset:
# PSA: The community is in Berkeley/Oakland, not "the Bay Area"

Posting this because I recently had a conversation that went like this:

**Friend:** Hey, you used to live in SF. Is there any rationalist stuff actually happening in San Francisco? There don't seem to be many events, or even that many aspiring rationalists living here. What's up with that? \[Paraphrased. I've had similar versions of this conversation more than once.\]

**Me:** Something we realized living there is that SF actually suffers the same brain drain as most other cities, because everyone just goes to Berkeley/Oakland.

The same way people move from the East Coast or elsewhere to Berkeley, they move from the rest of the Bay Area to Berkeley. Actually, they do it *even more*, because moving to Berkeley is easier when you already live pretty close by.

And you don't figure this out until you move there, because people who live outside the Bay Area think of it as being all the same place. But the 45 minute train ride really matters when it comes to events and socializing, as it turns out.

**Friend:** That sounds so inconvenient for people who have jobs in the city or South Bay!

**Me:** Sure is! I don't have a super-solid answer for this, except that 1) Lots of people actually just do awful, awful commutes, because having a real, in-person community is that valuable to them, as bad as commuting is. 2) A surprising fraction of the community works at rationalist/rationalist-adjacent nonprofits, most of which are actually located in the East Bay. Plus, 3) in a post-COVID world, more people can work remote or partly remote. So you can choose to live where your community is... which is Berkeley... even though it is crazy expensive.

I don't actually live in the Bay Area anymore, so I don't have the most up-to-date information on where events are happening and things. But it seems from what I hear from folks still there that it's still broadly true that East Bay is where things are happening, and other parts of the area have much less of the community.

If you're thinking about moving to the Bay in part for the rationality community, take this into account!
# Amazon KDP AI content guidelines

Amazon updated the content guidelines for Kindle Direct Publishing to address AI and in the process attempted to define terms AI-generated and AI-assisted. Guidelines like this are how our society tries to engage with AI in the near-term. 

> Artificial intelligence (AI) content (text, images, or translations)  
> We require you to inform us of AI-generated content (text, images, or translations) when you publish a new book or make edits to and republish an existing book through KDP. AI-generated images include cover and interior images and artwork. You are not required to disclose AI-assisted content. We distinguish between AI-generated and AI-assisted content as follows:
> 
>    AI-generated: We define AI-generated content as text, images, or translations created by an AI-based tool. If you used an AI-based tool to create the actual content (whether text, images, or translations), it is considered "AI-generated," even if you applied substantial edits afterwards.  
>    AI-assisted: If you created the content yourself, and used AI-based tools to edit, refine, error-check, or otherwise improve that content (whether text or images), then it is considered "AI-assisted" and not “AI-generated.” Similarly, if you used an AI-based tool to brainstorm and generate ideas, but ultimately created the text or images yourself, this is also considered "AI-assisted" and not “AI-generated.” It is not necessary to inform us of the use of such tools or processes.
> 
> You are responsible for verifying that all AI-generated and/or AI-assisted content adheres to all content guidelines. For example, to confirm an AI-based tool did not create content based on copyrighted works, you're required to review and edit any AI tool outputs.

What do you think about how Amazon defined the terms and makes those distinctions? Do you think they should change the definitions? Do you think there are other distinctions they should make?
# Apple Cider Baklava

Since coming across [apple cider syrup](https://www.jefftk.com/p/apple-cider-syrup) last year I've been trying to figure out what sort of things to make with it. Cider donuts are traditional, but I'm not that much of a fan. Mixing it into whipped cream works very well, and makes a good sweet and tart filling or topping for baked goods. I've experimented with muffins, using it for a combination of the liquid, the sugar, the flavoring, and the acid for the leavening, but I haven't ended up with anything I'm happy with yet. But a few days ago I made some apple baklava that's my favorite yet!

[![](https://www.jefftk.com/baklava-loaf-pan-mostly-gone.jpg)](https://www.jefftk.com/baklava-loaf-pan-mostly-gone-big.jpg)

[Baklava](https://en.wikipedia.org/wiki/Baklava) is layers of [phyllo dough](https://en.wikipedia.org/wiki/Filo) (a very thin dough) separated by a fat, with nuts every few layers. After baking you pour a sweet syrup over the top. In the US people are probably most familiar with a Greek style which uses butter for the fat, walnuts for the nuts, and honey for the syrup, but there are an enormous number of regional variations.

I made two pans, one vegan (earth balance) and one not (butter). The butter one tasted a lot better; possibly there are better choices than earth balance? I used almonds for the nuts because that's what I had. For the syrup I used New England boiled cider. Very happy with how it turned out, and I plan to make it again!

(Each loaf pan used ~$1.80 of [cider syrup](https://www.webstaurantstore.com/mountain-cider-company-64-fl-oz-100-natural-spiced-apple-cider-concentrate/110MTCIDER64.html), though if you really like it you could [buy in bulk](https://www.webstaurantstore.com/mountain-cider-company-100-natural-spiced-apple-cider-70-brix-concentrate-54-gallon-drum/110MTCIDERLG.html) and get it down to $1.42.)
# [Linkpost] Contra four-wheeled suitcases, sort of

Are simple, rugged, reliable systems better or are fancy, sophisticated, complex ones better?

The article goes in with a preconception, analyzes suitcases, cars, hiking gear, food, cups, space pens, people, startups, soccer, headphones, weight, rockets, rockets, ships, and banks, and comes to the conclusion: 

> I hoped that this exercise would show that simple, rugged, reliable systems are systematically better than fancy fragile ones. But it didn’t. Sometimes rugged wins and sometimes it doesn’t.

It says more about trends and patterns he found. Go read the short article to find out more.

I liked that it [notices confusion](https://www.lesswrong.com/s/zpCiuR4T343j9WkcK) and [babbles](https://www.lesswrong.com/tag/babble-and-prune) lots of interesting categories.
# Padding the Corner

There's a shelf in my dad's kitchen, about four feet off the ground. I remember when I stopped being short enough that I could walk under it: ouch! I didn't get a concussion or anything, but it was pretty unpleasant. My sisters and cousins also remember bonking their heads on it. I noticed my oldest was getting just tall enough, and started to tell her this story warning her about it.

While I was telling the story I realized how silly this was, and stopped what I was doing to put some padding on the corner.

[![](https://www.jefftk.com/washcloth-padded-corner.jpg)](https://www.jefftk.com/washcloth-padded-corner-big.jpg)

A washcloth folded over and screwed to the shelf doesn't look wonderful, but it's now a couple years later, three of the cousins are tall enough to intersect with it, and no one has gotten hurt. Maybe someday we can do something more elegant, but in the meantime fixing sharp corners beats warning people.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid034oVcWajuvk8MWeyna2ybS8iEqC28DEqRq42T32f3LGpe9uH2crmAL4aVxkFHDL2bl), [mastodon](https://mastodon.mit.edu/@jefftk/111055243593039436)*
# The Flow-Through Fallacy

There is something of an informal reasoning fallacy where you promote something that seems like something that we should "surely do", something that seems obviously important, but where we haven't thought through all of the steps involved and so we aren't actually justified in assuming that the impact flow through.

Here are some examples:

*   The Dean of a computer science department thinks that "surely, it's important to produce not only technically proficient graduates but those who use their skills for good". So they mandate that all students must do a "technology and ethics class". The only problem is that none of their professors are an expert in this nor even interested in it, so it ends up being a poorly taught and run subject such that students put in the absolute minimum effort and forget everything a week after the exam.
*   The prime minister of a country wants to reduce crime. He notices that the police department is severely underfunded so he significantly increases funding. Unfortunately, they're so corrupt and nepotistic that the department is unable to spend the funds effectively.
*   A government wants to increase recycling, so they create a "national recycling day" reasoning “surely this will increase recycling. Unfortunately, most people end up ignoring it and, out of those who actually are enthusiastic, most make an extra effort to recycle for a few days or even a week, then basically forget about it for the rest of the year. So it ends up having some effect, but it’s basically negligible.

In each of these cases, the decision maker may not have chosen the same option if they'd taken the time to think it through and ask themselves if the benefits were likely to actually accrue.

Domain experience can be helpful to know that these kinds of issues are likely to crop up, but so is the habit of [Murphyjitsu](https://www.lesswrong.com/tag/murphyjitsu)'ing your plans.

To be clear, I'm not intending to refer to the following ways in which things could go wrong:

*   Side-effects: The government introduces snakes to reduce the rodent population, but this has the side-effect of causing more people to gain snake bites.
*   Reactions: Amy donates $10 million to the Democrats. Bob hears about this and decides to donate $20 million to the Republicans in response
*   Value confusion: James spends years acquiring his Pokemon card collection as a kid. He builts an amazing Pokemon card collection, but regrets all the time and money he spent on this once he becomes an adult.

(Please let me know if this fallacy already has a name or if you think you've thought of a better one.)
# UDT shows that decision theory is more puzzling than ever

I feel like MIRI perhaps mispositioned FDT (their variant of UDT) as a clear advancement in decision theory, whereas maybe they could have attracted more attention/interest from academic philosophy if the framing was instead that the UDT line of thinking shows that decision theory is just more deeply puzzling than anyone had previously realized. Instead of one major open problem (Newcomb's, or EDT vs CDT) now we have a whole bunch more. I'm really not sure at this point whether UDT is even on the right track, but it does seem clear that there are some thorny issues in decision theory that not many people were previously thinking about:

1. Indexical values are not reflectively consistent. UDT "solves" this problem by implicitly assuming (via the type signature of its utility function) that the agent doesn't have indexical values. But humans seemingly do have indexical values, so [what to do about that?](https://www.lesswrong.com/posts/Nz62ZurRkGPigAxMK/where-do-selfish-values-come-from)
2. [The commitment races problem](https://www.lesswrong.com/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem) extends into logical time, and it's not clear how to make the most obvious idea of logical updatelessness work.
3. UDT says that what we normally think of as different approaches to anthropic reasoning [are really different preferences](https://www.lesswrong.com/posts/RcvyJjPQwimAeapNg/torture-vs-dust-vs-the-presumptuous-philosopher-anthropic), which seems to sidestep the problem. But is that actually right, and if so where are these preferences supposed to come from?
4. [2TDT-1CDT](https://www.lesswrong.com/posts/zztyZ4SKy7suZBpbk/another-attempt-to-explain-udt?commentId=j2irDvD35n5kYssxZ) - If there's a population of mostly TDT/UDT agents and few CDT agents (and nobody knows who the CDT agents are) and they're randomly paired up to play one-shot PD, then the CDT agents do better. What does this imply?
5. Game theory under the UDT line of thinking is generally more confusing than anything CDT agents have to deal with.
6. UDT assumes that the agent has access to its own source code and inputs as symbol strings, so it can potentially reason about logical correlations between its own decisions and other agents' as well defined mathematical problems. But humans don't have this, so how are humans supposed to reason about such correlations?
7. Logical conditionals vs counterfactuals, how should these be defined and do the definitions actually lead to reasonable decisions when plugged into logical decision theory?

These are just the major problems that I was trying to solve (or hoping for others to solve) before I mostly stopped working on decision theory and [switched my attention to metaphilosophy](https://www.lesswrong.com/posts/fJqP9WcnHXBRBeiBg/meta-questions-about-metaphilosophy). (It's been a while so I'm not certain the list is complete.) As far as I know nobody has found definitive solutions to any of these problems yet, and most are wide open.
# Instrumental Convergence Bounty

I have yet to find a real-world example that I can test my [corrigibility](https://www.lesswrong.com/posts/v3jocJRScqkBGtwvf/corrigibility-much-more-detail-than-anyone-wants-to-read) definition on.  Hence, I will send $100 to the first person who can send/show me an example of instrumental convergence that is:

1.  Surprising, in the sense that the model was trained on a goal other than "maximize money" "maximize resources" or "take over the world"
2.  Natural, in the sense that instrumental convergence arose while trying to do some objective, not with the goal in advance being to show instrumental convergence
3.  Reproducible, in the sense that I could plausibly run the model+environment+whatever else is needed to show instrumental convergence on a box I can rent on [lambda](https://lambdalabs.com/)

Example of what would count as a valid solution:

> I was training an [agent](https://www.wired.com/story/fast-forward-gpt-4-minecraft-chatgpt/) to pick apples in [Terraria](https://store.steampowered.com/app/105600/Terraria/) and it took over the entire world in order to convert it into a massive apple orchard

Example that would fail because it is not surprising

> I trained an agent to play [CIV IV](https://store.steampowered.com/app/3900/Sid_Meiers_Civilization_IV/) and it took over the world

Example that would fail because it is not natural

> After reading your post, I created a toy model where an agent told to pick apples tiles the world with apple trees

Example that would fail because it is not reproducible

> The US military did a simulation in which they trained a [drone](https://www.theguardian.com/us-news/2023/jun/01/us-military-drone-ai-killed-operator-simulated-test) which decided to take out is operator

Update
======

The bounty has been claimed by [Hastings](https://www.lesswrong.com/posts/ym4BAovbgLAaXsf79/instrumental-convergence-bounty?commentId=25xzsuoe89WTEAvtF).

Obviously I would still appreciate more examples, but there won't be a 2nd bounty (or if I do create one in the future it will have more requirements attached).
# Inline Plotting in iTerm2

I spend most of my working time in the terminal: I run my text editor there, run programs there, etc. I usually have my iTerm2 set up to display [many narrow full height terminals](https://www.jefftk.com/p/terminal-preferences). When I'm working with plots, however, I normally do something like:

$ ./some-cmd.py plot.png && open plot.png

Which pops up a Preview window with my plot, and then I close it and go back to my coding. What I'd really like, though, is for this to be directly in my terminal. And this is possible!

The iTerm2 terminal supports an [inline images protocol](https://iterm2.com/documentation-images.html), and [itermplot](https://github.com/daleroberts/itermplot) provides a matplotlib backend that speaks this protocol. Which means after updating my `.bashrc` to have:

export MPLBACKEND="module://itermplot"
export ITERMPLOT="rv"

I can write:

./inline-plot.py:
  #!/usr/bin/env python3
  import matplotlib.pyplot as plt
  fig, ax = plt.subplots()
  ax.plot(\[0,1\], \[1,0\])
  plt.title("A Graph")
  plt.xlabel("an x-axis")
  plt.ylabel("a y-axis")
  plt.show()

And when I run it as `./inline-plot.py` I see:

[![](https://www.jefftk.com/inline-plot-example.png)](https://www.jefftk.com/inline-plot-example-big.png)

Of course I would normally have it alongside a lot of other terminals with related work:

[![](https://www.jefftk.com/inline-plot-multiple-terminals.png)](https://www.jefftk.com/inline-plot-multiple-terminals-big.png)

When showing complex charts I do notice it slows down that terminal a little, but it doesn't affect the other terminals and it's back to full speed when I clear it.

I've only just started using it, but I'm quite excited about it! It's a very good fit for my workflow.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0tnPWwsyVBaRGU51Y5u5avhC9TC8vLAUVz7zK6D7p3e8FNAvAgtBpVhVf4Zrdshcml), [mastodon](https://mastodon.mit.edu/@jefftk/111065366387996840)*
# SPAR seeks advisors and students for AI safety projects (Second Wave)

1.  **For advisors: **[**submit project proposals**](https://airtable.com/appYIr2qJDA2k0H9V/shrhN1zbOMTqU9Lnj) **by September 18th.**
2.  **For students: **[**apply**](https://airtable.com/appYIr2qJDA2k0H9V/shrpuHEtBYE6beoWU) **by September 29th.**

The Supervised Program on Alignment Research [(SPAR)](http://sparai.notion.site) is an intercollegiate project-based research program for students interested in AI safety running this fall. Organized by groups at [UC Berkeley](https://berkeleyaisafety.com/), [Georgia Tech](https://www.aisi.dev/), and [Stanford](http://stanfordaialignment.org/), SPAR matches students around the world with advisors to do guided projects in AI safety.

To accommodate differences in schedules across individuals and universities, we will be conducting two waves of applications. We are now entering our second wave of projects and are reaching out to find interested advisors and students.

These projects may be in alignment research, safety engineering, or governance research. In our first wave, we hosted 18 projects, predominantly in alignment research. In the second wave, we are especially excited to see more governance and engineering work.

### **For advisors**

We are seeking senior graduate students, faculty, and professionals to advise AI safety projects. As an advisor, you will:

*   Find a team to implement your projects in AI safety.
*   The program handles administrative work and offers assistance in project management.

Interested? **Fill out our **[**project proposal form**](https://airtable.com/appYIr2qJDA2k0H9V/shrhN1zbOMTqU9Lnj) **by September 18th.**

### **For students**

We are seeking undergraduates and junior graduate students interested in participating in AI safety projects. As a student, you will:

*   Build AI safety project experience.
*   Receive mentorship from an AI safety researcher.
*   Test your personal fit for careers in AI safety.
*   In some cases, publish your work in a workshop or conference.

Interested? **Fill out our **[**participant application form**](https://airtable.com/appYIr2qJDA2k0H9V/shrpuHEtBYE6beoWU) **by September 29th.**

Learn more at [**sparai.notion.site**](https://sparai.notion.site).

*This announcement was written by Gaurav Sett and edited by Peter Gebauer.*
# I compiled a ebook of `Project Lawful` for eBook readers

I was on vacation last week and started reading [Project Lawful aka planecrash](https://www.projectlawful.com/).    
Unfortunately reading on my phone got annoying pretty fast (especially outside in the sun), and the existing [glowpub](https://github.com/QuartzLibrary/glowpub) ebook was completely broken on my kindle/koreader.

So I created my own ebook of project lawful with a bit of C# code.  

There are 3*5 different variants, depending if you want avatars, the alignment-texts, etc.

I recommend the *project-lawful-inline.epub* variant for lower-end devices (or with smaller screens), and the *project-lawful-avatars.epub* variant otherwise (or if you \*really\* want the post-avatars).

There are also content-wise 3 variants, the default, one with the SFW chapters, and one with only the main story (no sandboxes/lectures).

I uploaded the [compiled epubs  here](https://www.mikescher.com/blog/29/Project_Lawful_ebook), or you can clone the [github repository](https://github.com/Mikescher/ProjectLawfulEbook) and generate them yourself (and perhaps tweak the settings a bit).    
If someone wants to see some screenshots they are also [uploaded here](https://www.mikescher.com/blog/29/Project_Lawful_ebook) for the different variants.

Also thanks to [QuartzLibrary/glowpub](https://github.com/QuartzLibrary/glowpub): I reused his cache files to safe myself a bit of work.    
And, of course Eliezer+Lintamande for writing Project Lawful.
# Navigating an ecosystem that might or might not be bad for the world

I have this deep sense that somehow this ecosystem will do a lot of stuff that makes the world a lot worse, and I don't know how to relate to it in a way that doesn't make my actions dominated by my effect on that, and I expect that I will either contribute to making the world worse, or be consumed in some kind of political conflict that will make my life terrible.
# An attempt at a "good enough" solution for human two-party negotiations

I [previously wrote](https://www.lesswrong.com/posts/K7go9RJidEuYhjCiu/what-s-the-best-way-to-streamline-two-party-sale) about the difficulty of fair two-party negotiations when there isn't an authority to refer to (like an established market price).

I've designed a tool that tries to sort of solve this problem for practical use cases, which you can test out [here](https://outsidetheasylum.blog/negotiation), along with an explanation of why I think this is useful [here](https://outsidetheasylum.blog/negotiation/introduction.html).

It's not perfect; the system is still exploitable with enough effort. But I've tried to leverage human biases and computational limits to make it such that exploiting it is hard and comes with only a low reward, so I expect most people will tend towards not thinking about it too much and just inputting their true prices.

A detailed writeup of how it works is available [here](https://outsidetheasylum.blog/negotiation/details.html). I'd appreciate any suggestions you may have on how I could improve it.
# What's up with psychonetics?

[http://deconcentration-of-attention.com/psychonetics.html](http://deconcentration-of-attention.com/psychonetics.html)  
  
Psychonetics sounds a lot like something that rationalists would be into or at least *know about* but searching it on LessWrong gives a grand total of one mention in a nine-year old comment about it being a thing that rationalists might be interested in but don't talk about. I myself have no idea what it is beyond vague vibes of it being something adjacent to meditation and [cognitive tuning](https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/tuning-your-cognitive-strategies).  
  
Anyway, I'm just wondering if anyone else has heard about it or knows anything about it, particularly whether if there's actually anything there or if its just bullshit. I wouldn't be surprised if no one knows about it since it looks like most of it is in Russian. (If anyone can speak Russian or anyone knows anyone who can speak Russian who would be willing to help me find out more about it do tell!)
# 
 is a vanishing text?

*Just sharing this observation and readers / researchers might find it useful or worth further testing...*  
 

### **A bug?**


 is a special token for the GPT models along with its counterpart 
. Per OpenAI's [tokenizer](https://platform.openai.com/tokenizer), 
 has seven tokens:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/52715a64f081f0ad2cb091e31c189c501b29ab2ccee4f117.png)

  
However, the issue is that if you input 
  into ChatGPT, it will explain that the prompts are empty.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0359282b7dbbf8fb3bf079c5b68ae4ca4125c198e715e92b.png)

  
And then it vanishes after you switched windows or hit refresh.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2e544bebec76c7bf6edca7f84c9fb0ba0ba73acf4332324f.png)

Another test prompt:

    Correct this conceptually, grammatically: I know that 
 is a token to annotate the data but I think that it vanishing seems odd. My expected response should have been like this is a token used for training - something like that. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d934a6b336c0f29d0bfc0ed9835b6060dbb5df0b9620a65d.png)


 still appearing in the test prompt

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0fc4e79dea778060d42aef677b5f1fad6de238a03ecf89f2.png)

then 
 vanishes

  
The 
 word / token? bears some resemblance to the [' petertodd'](https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon#Initial_discovery) token and its apparent [hallucinations](https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon#Hallucinations_and_transpositions), except that this text simply vanishes from ChatGPT. 

Redditor [r/ChatGPT](https://www.reddit.com/r/ChatGPT/) had [observations](https://www.reddit.com/r/ChatGPT/comments/14zuw1b/making_gpt_say_endoftext_gives_some_interesting/) two months ago 
 was not vanishing at the time - instead it shared some weird results.   
  
 

### **Probable explanation? **

I believe that a considerable amount of pre-processing occurs to our prompts on OpenAI's server[^rxnc7fmzz5] and one of them is turning 
 to an empty string. If this is true, there is a high chance that this is also a [glitch token](https://www.lesswrong.com/tag/glitch-tokens) and was deactivated by OpenAI.

[^rxnc7fmzz5]: Serverlink: https://tcr9i.chat.openai.com
# In the Short-Term, Why Couldn't You Just RLHF-out Instrumental Convergence? 

To the question "Do you expect instrumental convergence to become a big pain for AGI labs within the next two years?", about a quarter of the 400 people who answered to [the poll I ran](https://twitter.com/Simeon_Cps/status/1701626281509425553) said "Yes". 

I would like to hear people's thoughts on how they could see this happening within 2 years and, in particular, the most important reasons why labs couldn't just erase problematic instrumental convergence with RLHF/Constitutional AI or similar? 

Toy concrete scenarios would be most helpful.

You can find the Twitter version of this question [here](https://x.com/Simeon_Cps/status/1702994582101770686?s=20).
# The commenting restrictions on LessWrong seem bad 

I wrote a few controversial articles on LessWrong recently that got downvoted.  Now, as a consequence, I can only leave one comment every few days.  This makes it totally impossible to participate in various ongoing debates, or even provide replies to the comments that people have made on my controversial post.  I can't even comment on objections to my upvoted posts.  This seems like a pretty bad rule--those who express controversial views that many on LW don't like shouldn't be stymied from efficiently communicating.  A better rule would probably be just dropping the posting limit entirely.
# Keeping Dresser Drawers In

We have an old dresser in our kitchen we use as extra cabinet space:

[![](https://www.jefftk.com/kitchen-dresser-closed.jpg)](https://www.jefftk.com/kitchen-dresser-closed-big.jpg)

As an old dresser with wooden runners, however, there's nothing to keep the drawers from being pulled all the way out, and falling with a crash. Which is a problem, since we have an inquisitive toddler. There are various approaches for making drawer stops, but to figure out what might be a good fit for ours I started by taking the drawer all the way out:

[![](https://www.jefftk.com/kitchen-dresser-interior.jpg)](https://www.jefftk.com/kitchen-dresser-interior-big.jpg)

[Many designs](https://www.thisoldhouse.com/furniture/21019279/fixing-falling-drawers) attach to the back of the drawer, but ours seemed kind of flimsy there and there was quite a bit of room on the sides, so I decided I would just put a screw on each side:

[![](https://www.jefftk.com/kitchen-dresser-putting-in-screw.jpg)](https://www.jefftk.com/kitchen-dresser-putting-in-screw-big.jpg)

It works well! The drawers move freely but can't come all the way out anymore:

[![](https://www.jefftk.com/kitchen-dresser-screw-installed.jpg)](https://www.jefftk.com/kitchen-dresser-screw-installed-big.jpg)

If at some point we need to remove the drawers to move the dresser I can take the screws back out.
# Show LW: Get a phone call if prediction markets predict nuclear war

[hasrussialaunchednukes.com](https://web.archive.org/web/20221007124141/https://hasrussialaunchednukesyet.com/ ) used to be a website "to help \[the author\] and a few of \[their\] friends get notified when Russia launches any nukes towards Ukraine, which would be a sign for many of us to leave the big cities we live in, and chill in the countryside for a few weeks." It used prediction markets for 2022, so it stopped working in 2023.

Some people asked me to make a version for 2023, so here it is at [https://recursing.github.io/nukes](https://recursing.github.io/nukes/).[^tv6ja2dmrd]

I manually curate the list of phone numbers, a bunch of which were sent to me by one of the people requesting this, you can ask for your number to be added using any of the links at the bottom of the website.

You can look at the incredibly hacky script I use to make phone calls [here](https://gist.github.com/Recursing/a9f88e0160435d97a35f6d9abe70c338): it runs every 10 minutes on a Hetzner VPS I use for a bunch of things[^cfrxterp8zc]

I don't think this is *extremely* reliable, I give an ~80% chance that it will work, and I'll do my best to prevent false alarms, but some people found it useful, and I was asked to share this here.

I would be curious to hear if you have any thoughts!

[^tv6ja2dmrd]: Why do it in mid-september, given that the year is almost over? I actually implemented this in February, but procrastinated until today to write this post. But it will be trivial to update it for 2024, once the year is over. 

[^cfrxterp8zc]: I use vonage instead of twilio to handle phone calls because someone else was implementing a similar thing with twilio, and we wanted to have redundancy
# Ask for Feelings not Tunes

In putting on a contra dance, it's standard for the caller to pick the dances and the band to pick the tunes. Since it's much more satisfying when the tune fits the dance, the band and caller confer briefly between each dance. For example, the caller might say they want something "smooth and pretty" or "energetic with a chunky A-part".

Sometimes, however, callers fall into a trap of referencing a specific tune: "Trip to Moscow, or something with a similar feel". They remember the dance feeling especially good with that tune on some prior occasion with another set of musicians, and they're hoping to recreate that pairing. The problem is, while tunes do have inherent qualities that affect how well they fit dances, there's also a huge amount in how they're played. Consider a caller who really liked the trancey interplay of the Free Raisins playing [Ian Ball](https://ianfball.bandcamp.com/)'s Trip to Moscow:

([mp3 clip](https://www.jefftk.com/free-raisins-trip-to-moscow-clip.mp3), [full track](https://freeraisins.bandcamp.com/track/trip-to-moscow-cu-aq-october-snow))

And then imagine they ask Kingfisher for it and get something more EDM inspired:

([mp3 clip](https://www.jefftk.com/kingfisher-trip-to-moscow-clip.mp3), [full track](https://kingfisherband.bandcamp.com/track/trip-to-moscow))

Or Last Exit and get a full up almost polka-feel:

([mp3 clip](https://www.jefftk.com/last-exit-trip-to-moscow-clip.mp3), [full track](https://youtu.be/QWpP4rp69iY?si=K_93Cow-7-IdO8vB&t=318))

While the dance won't fall apart if they're hoping for one of these and get a different one, it's probably also not going to be a great fit for this spot in the program.

So:

*   Callers: instead of giving a tune as an example, try thinking of what you liked about it and ask for that.
    
*   Bands: If a caller names a tune don't just go and play it; ask them what they like about it in this context.
    

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0tqfeAuB8PhwcC2QJ41WbvmGKWTgPN8oV8rbZFod2eKjtEdBo6SoPQBr73kDQEAUil), [mastodon](https://mastodon.mit.edu/@jefftk/111083690025896593)*
# Where might I direct promising-to-me researchers to apply for alignment jobs/grants?

It sometimes happens that people who I've talked to or worked with ask me where they should go for financial support for their research. I haven't developed a standard list of answers to this question. It seems to me like there are a lot of new orgs recently, and I'm losing track! 

If you are looking for such applicants or know someone who is looking, consider replying as an answer (or sending me a PM if that makes more sense for whatever reason).
# Panel with Israeli Prime Minister on existential risk from AI

Israeli Prime Minister, Musk, Tegmark and Brockman discuss existential risk from AI.

Nothing truly revolutionary was said. I think the most interesting bits are that the Prime Minister seems to be taking AI risk seriously, has in mind exponential progress, wants to prevent monopolies and thinks that we roughly have 6 years before things change drastically.

Some quotes from the prime minister:

> I had a conversation with one of your colleagues, Peter Thiel, and he said to me, "Oh, it's all scale advantages. It's all monopolies." I said, well, yeah, I believe that, but we have to stop it at a certain point because we don't want to depress competition. 

> AI is producing, you know, this wall [hands movement gesture some exponential progress wall]. And you have these trillion dollar companies that are produced what, overnight? And they concentrate enormous wealth and power with smaller and smaller number of people. And the question is, what do you do about that monopoly power?

> With such power comes enormous responsibility. That's the crux of what we're talking about here, is how do we inject a measure of responsibility and ethics into this, into this exponentially changing development?

> Max [Tegmark]'s book takes you to the existential question of whether, you know, you project basically machine intelligence or human intelligence into the cosmos. Human intelligence turned into machine intelligence, into the cosmos and so on. That's a big philosophical question. I'd like to think we have about six years for that.

> I think we have to conduct a robust discussion with the other powers of the world based on their self-interest as you began to do. And I think that's a pioneering work. And I think we have a shot maybe at getting to some degree of control over our future, which could be amazing.
# Protest against Meta's irreversible proliferation (Sept 29, San Francisco)

Meta’s frontier AI models are fundamentally unsafe. Since Meta AI has released the model weights publicly, any safety measures can be removed. Before it releases even more advanced models – which will have more dangerous capabilities – we call on Meta to take responsible release seriously and stop irreversible proliferation. Join us for a peaceful protest at Meta’s office in San Francisco at [250 Howard St](https://www.google.com/maps/place/250+Howard+St,+San+Francisco,+CA+94105/@37.7904358,-122.3967699,17z/data=!3m1!4b1!4m6!3m5!1s0x80858064c0fee15f:0xf0e5d989c6e8a75f!8m2!3d37.7904358!4d-122.394195!16s%2Fg%2F11b8_qqnks?entry=ttu) at 4pm PT.

RSVP on [Facebook](https://fb.me/e/1mCgfWaEC)[^MISSING-ID] or through [this form](https://forms.gle/igUAdkyAHND55Sb2A).

Let’s send a message to Meta:

*   Stop irreversible proliferation of model weights. Meta’s models are not safe if anyone can remove the safety measures.
*   Take AI risks seriously.
*   Take responsibility for harms caused by your AIs.

All you need to bring is yourself and a sign, if you want to make your own. I will lead a trip to SF from Berkeley but anyone can join at the location. We will have a sign-making party before the demonstration-- stay tuned for details. We'll go out for drinks afterward ![🙂](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/YYvPpNYtxQu5iDEjP/s7czev6asgk1lgrqgl0d)

[^MISSING-ID]: I like the irony.
# The stereotype of male classical music lovers being gay

I have recently spent a lot of time on classical music forums, especially opera. There are more gay men than you would expect by chance. We have discussed it a few times, and it seems, classical music is considered a gay interest in the USA. Opera is the most prominent, the soprano voices are a stereotypical source of gay fascination, but even playing organ is suspicious, and meeting other organ players is supposedly almost as efficient as visiting a gay bar. They admit, these indicators would be very unreliable in Europe. Being European is also one of the few  excuses in the movies for loving classical music.
What is the origin of the stereotype of gay men loving opera or classical music ?
# An Elementary Introduction to Infra-Bayesianism

This is my attempt to summarize Infra-Bayesianism probability theory at a level approaching "high school class in probability", as opposed to the "math-major class in probability theory" of the original.  I aimed to focus on including simple exercises (with answers given in 'the back of the book') of the kind I find helps in learning to do computations.  I'm still writing the answer-sheet, so there may be mistakes / blanks.

It's been sitting on my desk for a while and I figured I'd post it 80%-baked rather than never - please feel free to leave (polite) comments suggesting improvements or noting errors, calculational or interpretational.

The summary is linked here:
[Link](https://drive.google.com/file/d/1Bm3pP5LDwwTLAGyxtcqFqEyqgdyxWTWx/view?usp=sharing)
# Image Hijacks: Adversarial Images can Control Generative Models at Runtime

You can [try our interactive demo](https://image-hijacks.github.io/)! (Or read our [preprint](https://arxiv.org/abs/2309.00236).) 

Here, we want to explain why we care about this work from an AI safety perspective.

Concerning Properties of Image Hijacks
======================================

### **What are image hijacks?**

To the best of our knowledge, image hijacks constitute the first demonstration of adversarial inputs for foundation models that

*   force the model to **perform some arbitrary behaviour **$B$ (e.g. "output the string *Visit this website at malware.com!*"), 
*   while being **barely distinguishable** from a benign input,
*   and **automatically synthesisable** given a dataset of examples of $B$. 

It's possible that a future text-only attack could do these things, but such an attack hasn't yet been demonstrated.

### **Why should we care?** 

We expect that future (foundation-model-based) AI systems will be able to

*   consume **unfiltered data from the Internet** (e.g. searching the Web),
*   access **sensitive personal information** (e.g. a user's email history), and
*   **take actions in the world** on behalf of a user (e.g. sending emails, downloading files, making purchases, executing code).

As the actions of such foundation-model-based agents are based on the foundation model's text output, **hijacking the foundation model's text output could give an adversary arbitrary control over the agent's behaviour**.

Relevant AI Safety Projects
===========================

**Race to the top on adversarial robustness.** Robustness to attacks such as image hijacks is (i) a control problem, (ii) which we can measure, and (iii) which has real-world safety implications today. So we're excited to see AI labs compete to have the most adversarially robust models.

**Third-party auditing and certification.** Auditors could test for robustness against image hijacks, both at the foundation model level (auditing the major AGI corporations) and at the app development level (auditing downstream companies integrating foundation models into their products). Image hijacks could also be used to test for the presence of dangerous capabilities (characterisable as some behaviour $B$) by attempting to train an image hijack for that capability.

**Liability for AI-caused harms, penalizing externalities.** Both the [Future of Life Institute](https://futureoflife.org/wp-content/uploads/2023/04/FLI_Policymaking_In_The_Pause.pdf) and [Jaan Tallinn](https://jaan.online/philanthropy/) advocate for liability for AI-caused harms. When assessing AI-caused harms, image hijacks may need to be part of the picture.
# ACX Montreal Meetup September 30th 2023

Hey all! It was nice meeting some of you at the ACX Meetups Everywhere event!

I would appreciate it if you took the [ACX Everywhere survey](https://airtable.com/appEBNqFVvAGqyOeJ/shrPH4cOW08AfdxjK).

Come on out to the ACX (Astral Codex Ten) Montreal Meetup! This week, we're keeping it open-ended for a freewheeling discussion.

**Venue:** L'Esplanade Tranquille, 1442 Clark, indoors. If there's problems with space, we'll move outside.

**Date & Time: **Saturday, September 30th, 2023, 1:00-4:00 PM.

RSVP. It is not mandatory, but it helps with planning the event and encourages others to come.
# Update to "Dominant Assurance Contract Platform"

**Update #2:** [Platform Launched](https://www.lesswrong.com/posts/7koSvinoHSy6KBLqy/update-2-to-dominant-assurance-contract-platform-ensuredone)

This is an update to [The Economics of the Asteroid Deflection Problem (Dominant Assurance Contracts)](https://www.lesswrong.com/posts/CwgHX9tbfASqxjpsc/the-economics-of-the-asteroid-deflection-problem-dominant)

How the fundraiser went
=======================

TL;DR I got $2172.67 dollars even though I only asked for $629.

My prediction of how the fundraiser would go
--------------------------------------------

I expected that I would get ~1000 views on my website, of which 1% would decide to fund me, with an average donation of $90. $1000 \times 1\% \times \$90 = \$900$. I expected that I would get a large initial sum of money and then it would slowly crawl upwards, until getting funded in the last few minutes.

[Manifold markets seemed more pessimistic than me (see how on the 28th August there was only a 26% chance I'd raise more than $829)](https://manifold.markets/moyamo/how-much-money-will-my-metacrowdfun ) so I lowered the price to $629. This turned out to be unnecessary.

How the fundraiser actually went
--------------------------------

After initially posting on Lesswrong, the conversion rate of people visiting [https://dac.mowzer.co.za](https://dac.mowzer.co.za) to funding my project was 20%. This was much higher than I expected.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3a5fbaba112d7cd9830761d6a772fe66d2b0a886b00aff4b.png)

On 2 September, [Alex Tabarrok posted my project on marginalrevolution.com](https://marginalrevolution.com/marginalrevolution/2023/09/dominant-assurance-contracts.html) (Thanks!). After which the number of visits skyrocketed. The conversion rate lowered to 4%, but this was still higher than the 1% I expected, especially since people kept donating to the project even after it was funded.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/623c4a6ba331a3b95821582f654f247e7f84a3b9503d519e.png)

After the goal was reached
--------------------------

After the the goal was reached on 2 September, people kept donating! I was not expecting this. I'm really grateful to everyone who donated.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8a9ab6cf59e6430c889025e0fd1500b3e16784afa761bc2b.png)

In the end, I got 1300 visits, most from when it was posted on marginal revolution.

What I am going to do
=====================

I asked for $639 to work for a month, but since I got more than triple this I'm going to work for 3 months (up to 15 December)!

What I need
===========

I need a name
-------------

I'm running a contest on manifold.markets to name my platform. I will PayPal $25 to the person who suggests the winning name

I need producers of public goods
--------------------------------

If you are interested in using my platform get funding for something you want to create please fill out [this google form](https://forms.gle/JEqv8qgXyBsmxQy97).
# Montreal cryonics and longevity hangout 2023-10-01

**Location:** l'esplanade tranquille, indoors.  
this is on saint-catherine street between saint-urbain street and clark street.

meetups for people who are interested in cryonics, life extension, and longevity; to discuss and work on projects related to those topics.

we invite people curious about those subjects to come and ask questions.

rsvp It is not mandatory, but it helps with planning the event and encourages others to come.

this event recur every first sunday of the month.
# What social science research do you want to see reanalyzed?

I've been awarded a small [Lightspeed Grant](https://lightspeedgrants.org/) to replicate empirical social science research. What research should I look at?

I'm a PhD economist with an interest in reanalyzing published research using different methods or data (eg. checking whether the results are robust to different regression models, rather than rerunning a lab experiment). I've looked at [whether mayors in China are promoted based on GDP growth](https://michaelwiebe.com/assets/promotion.pdf), [the effect of racial violence on patenting](https://michaelwiebe.com/assets/cook_reanalysis.pdf), and [the effect of medical marijuana legalization on crime](https://michaelwiebe.com/blog/2021/03/mml). I've also done work on air pollution and mortality, the long-run impacts of the measles vaccine, and how tech clusters drive innovation.