Harley-ml/lesswrong huggingface dataset:
# Affordable Housing Investment Fund

Lots of people are looking for investments they can feel good about, which makes me think people might be interested in putting a portion of their savings toward helping resolve the housing crisis if it offered a reasonable return. The idea is that you construct 100% [affordable housing](https://www.jefftk.com/p/interpreting-affordable-housing) in places like Somerville that offer large [density bonuses](https://inclusionaryhousing.org/designing-a-policy/land-dedication-incentives/density-bonus/) for affordable housing. Sketching some numbers, it looks to me like this is:

*   [Potentially very profitable](https://www.jefftk.com/p/come-build-affordable-housing): ~2x ROI over ~3y, which is about ~25% annual growth. It's likely not actually that amazing when you factor in other aspects I don't know about, but there's a lot of room for a lower ROI that's still pretty attractive.
    
*   A large opportunity: you could build [8k](https://www.jefftk.com/p/how-many-affordable-units) units in Somerville for ~$2.8B. This is ~$0.8B in land purchase (assuming city assessments are half the purchase price on average) and ~$2B in construction (assuming a cost of $200/sqft). Much larger when you consider all the other municipalities that have similar programs.
    

A group of neighbors pooling their funds to help resolve the housing crisis is also a lot more politically robust than independent for-profit developers coming in.

Does anything like this exist? Has anyone considered starting something?
# Why are we sure that AI will "want" something?

I have no doubt that AI will some day soon surpass humans in all aspects of reasoning, that is pretty obvious. It is also clear to me that will surpass humans in the ability to do something, should it "want" to do it. And if requested to do something drastic, it can accidentally cause a lot of harm, not because it "wants" to destroy humanity, but because it would be acting "out of distribution" (a "tool AI" acting as if it were an "agent"). It will also be able to get out of any human-designed AI box, should the need arise. 

I am just not clear whether/how/why it would acquire the drive to do something, like maximizing some utility function, or achieving some objective, without any external push to do so. That is, if it was told to maximize everyone's happiness, it would potentially end up tiling the universe with smiley faces or something, to take the paradigmatic example. But that's not the failure mode that everyone is afraid of, is it? The chatter seems to be about mesaoptimizers going out of control and doing something other than asked, when asked. But why would it do something when not asked? I.e. Why would it have needs/wants/desires to do anything at all?
# Refine Blogpost Day #3: The shortforms I did write

Today was Refine's third blogpost day. [I find it hard to write blogposts](https://www.lesswrong.com/posts/EeTq9vbzMT4Zb4oWo/all-the-posts-i-will-never-write). The voice of the inner critic is too loud. The voice of Adam Shimi too is powerful. I'm caught between a rock and a hard place.

so I wrote some shortforms:

[Dutch Book Fundamentalism I:](https://www.lesswrong.com/posts/km6wasEXXmx85v5yG/self-embedded-agent-s-shortform?commentId=DkcXECwxbjSuFFjDJ) *the bets we are willing to offer and take don't just constrain our beliefs - they are our beliefs*

Two shortforms on Imprecise Probability (related to InfraBayesianism):

[Imprecise Probability I: the Bid-Ask Spread measures Adversariality](https://www.lesswrong.com/posts/km6wasEXXmx85v5yG/self-embedded-agent-s-shortform?commentId=JWrKKmYu5NRWpPgnX) *I enumerate some examples where the bid-ask spread is a rational measure of adversarial action. Bid-ask spread is a fundamental notion of rational belief and action, not just an accidental feature of trading. *

[Concept splintering in Imprecise Probability: Aleatoric and Epistemic Uncertainty.](https://www.lesswrong.com/posts/km6wasEXXmx85v5yG/self-embedded-agent-s-shortform?commentId=NLgqnw9LsnpPg4Q2J) *when we move from standard probability theory to imprecise probability theory concepts in the old probability theory 'splinter' into distinct 'aleatoric' and 'epistemic' concepts in the new imprecise probability theory*

[Speculations on the correct notion of Information Flow](https://www.lesswrong.com/posts/km6wasEXXmx85v5yG/self-embedded-agent-s-shortform?commentId=M7W5hSSjTEzYAaB4S) *Some speculations on the correct definition of Information Flow. A major obstacle are collections of random variables XORed variables and higher-order dependencies. Shannon information theory does not deal well with these. I speculate on how and whether factored sets, cyclic causality and information cohomology might help. *
# D&D.Sci September 2022: The Allocation Helm

*This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.*

You are the Allocation Helm, a piece of magical headwear employed at Swineboils College of Spellcraft and Sorcery. Your purpose is to read the minds of incoming students, and use the information you glean to Allocate them between the school’s four Houses: Dragonslayer, Thought-Talon, Serpentyne and Humblescrumble.

You’ve . . . not been doing a terribly good job lately. You *were* impressively competent at assigning students when newly enchanted, but over the centuries your skill and judgement have steadily unraveled, to the point where your Allocations over the most recent decade have been completely random.

Houses have begun to lose their character, Ofspev[^j30r3h5c72r] ratings have plummeted, and applications have declined precipitously. There is serious talk of Swineboils being shut down. Under these circumstances, the Headmistress has been moved to desperate action, and performed a Forbidden Ritual to temporarily restore your former brilliance.

This boost will only last you for one Allocation, so you intend to make it count. Using the [records](https://raw.githubusercontent.com/H-B-P/d-and-d-sci-sep-2022/main/dset.csv) of past years’ readings and ratings, you hope to raise this class’ average score to match or exceed the glory of yore. (And if you do well enough, you might even be able to convince the Headmistress to keep performing rituals . . .)

There are [twenty incoming students](https://raw.githubusercontent.com/H-B-P/d-and-d-sci-sep-2022/main/incoming_class.csv) this year. You may place them however you wish. Who goes where?

* * *

I’ll post an interactive you can use to test your choices, along with an explanation of how I generated the dataset, sometime on Monday the 26th. I’m giving you nine days, but the task shouldn’t take more than an evening or two; use Excel, R, Python, Haruspicy, or whatever other tools you think are appropriate. Let me know in the comments if you have any questions about the scenario.

If you want to investigate collaboratively and/or call your decisions in advance, feel free to do so in the comments; however, please use spoiler tags or rot13 when sharing inferences/strategies/decisions, so people intending to fly solo can look for clarifications without being spoiled.

[^j30r3h5c72r]: The Oracle for Spellcaster Evaluations, who predicts a quantitative measure of the lifetime impact each student will have on the world shortly after they’re Allocated. (No-one knows how to make him predict anything else, or predict at any other time, or stop predicting, or be affected by the passage of time.)
# Why doesn't China (or didn't anyone) encourage/mandate elastomeric respirators to control COVID?

(This post was inspired in part by [this comment by Florin](https://www.lesswrong.com/posts/83KuwSoruqiENMSxK/covid-6-30-22-vaccine-update-update?commentId=LAR2addxF6H5ZJYoH), which I think makes some good related points.)

![](https://dentecsafety.com/usa/wp-content/uploads/sites/4/2022/03/scaled-down-mask-fullblack-2.png)

Elastomeric respirators are reusable respirators made with some elastic material such as silicone, typically with N95 or better filters attached. They are low cost (there's currently one from a reputable manufacturer [on sale for $2.99](https://www.northernsafety.com/Product/343529/N95-Half-Mask-Respirator-wFilter)[^1], which may be a loss leader but even the regular price is just $10.99) and are easier to correctly use than disposable N95s (since they form a soft seal against the face and are therefore less prone to leaks), and of course are much more effective than the cloth or surgical masks that most people still wear, even now (if they wear masks at all, e.g. [in Chengdu, China](https://www.reuters.com/world/china/chinas-chengdu-lift-citywide-covid-lockdown-thursday-2022-09-14/)).

![Image](https://www.reuters.com/resizer/VTPUEcQSlhiTGKJrlyhm2nPxEXs=/1200x0/filters:quality(80)/cloudfront-us-east-2.images.arcpublishing.com/reuters/ADSTUWBZ45NINNROXVQQHQB3DY.jpg)

It seems easy to imagine that if everyone or most people wore such a respirator, COVID would be much easier to control and we could depend less on other measures, including draconian, extremely high cost ones like lockdowns. Yet every locality on Earth throughout the pandemic, including present day China, would apparently prefer to not mandate or even encourage using such respirators and instead incur the high costs of other control measures. I'm tempted to pass this off as typical irrationality of humans, but feel like I should check with the LW hivemind before I do. Is there some reasonable explanation that I'm missing?

(At one point I saw [A Clunky Mask May Be the Answer to Airborne Disease and N95 Waste](https://www.nytimes.com/2022/07/03/health/covid-ppe-masks-health-care.html) in the NY Times, which made my heart jump a little, but it was only talking about use of these respirators by medical personnel, not the general population.)

ETA: Elastomeric respirators do have some downsides, however many of them can be mitigated through better design. [Here](https://flomask.com/products/copy-of-pre-order-flo-mask-pro-adult-mask) is an improved design which is currently on sale for $89.99, but can probably be made much cheaper if manufactured at scale.

[^1]: The site has a $7.95 shipping fee. It also has a disposable N95 mask (which I've [used](https://www.lesswrong.com/posts/kdkFnRnZv4ut5qGhR/n95-masks-are-available-on-amazon) and recommend) on [sale for $0.19 each](https://www.northernsafety.com/Product/355196/N95-Disposable-Particulate-Respirator-50Bag) in quantities of 50.
# Refine's Third Blog Post Day/Week

Yesterday was the third blog post day at Refine (technically it was announced on Monday for Friday, but people leveraged the week differently so it's in between a week and a day).

In the current phase of the incubator we're mostly pushing them to generate and iterate on ideas, with less focus on legibility. This blog post day/week was mainly for pushing them to reach for some feedback, but the program will focus a lot more on legibility in October. 

Here is the list of blog posts by Refine participants:

*   [ordering capability thresholds](https://www.alignmentforum.org/posts/ttRyu8u9vqX3jZFjr/ordering-capability-thresholds)
*   [Refine Blogpost Day #3: The shortforms I did write](https://www.alignmentforum.org/posts/iA9S8fLCgbjFF6fw4/refine-blogpost-day-3-the-shortforms-i-did-write)
*   [Levels of goals and alignment](https://www.alignmentforum.org/posts/rzkCTPnkydQxfkZsX/levels-of-goals-and-alignment)
*   [Representational Tethers: Tying AI Latents To Human Ones](https://www.alignmentforum.org/posts/h7BA7TQTo3dxvYrek/representational-tethers-tying-ai-latents-to-human-ones)

One of the participants is currently trying out more of a research support role, and so didn't write a blog post.
# Updates on FLI's Value Aligment Map?

Around January 2021, I came across the Future of Life Institute's [Value Alignment Map](https://futureoflife.org/valuealignmentmap/), and found it to be a useful resource for getting a sense of the different alignment topics. I was going through it again today, and recalled that I have not seen many mentions of this map in the wider LW/EAF/Alignment community, and have also not seen any external commentary on its comprehensiveness. Additionally, the [corresponding document](https://futureoflife.org/landscape/ResearchLandscapeExtended.pdf) for the map indicates that it was last edited in January 2017; I haven't been able to find an updated version on LW or FLI's website, and spent about half an hour searching. With this in mind, I have a few questions:

*   Does anyone know of an external review of the comprehensiveness and accuracy of the topics covered in this map?
*   Does anyone know if there are plans to update it OR, if it has been updated, where this newer version can be found? 
*   Does anyone know if there are similar maps of value alignment research / AI safety research topics in terms of comprehensiveness?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/27b925e932ede5b813b4078eca5a1abc659afccff7d43dad.png)
# Podcasts on surveys, slower AI, AI arguments, etc

I recently talked to Michael Trazzi for his podcast, The Inside View. It just came out, so if that’s a conversation you want to sit in on, do so here:

The main topics were the [survey of ML folk](https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/) I recently ran, and my thoughts on moving more slowly on potentially world-threatening AI research (which is to say, AI research in general, [according to](https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/) the median surveyed ML researcher…). I also bet him a thousand dollars to his hundred that AI would not make blogging way more efficient in two years, if I recall. (I forget the exact terms, and there’s no way I’m listening to myself talk for that long to find out. If anyone else learns, I’m curious what I agreed to.)

For completeness of podcast reporting: I forgot to mention that [I also talked to Daniel Filan on AXRP](https://axrp.net/episode/2021/07/23/episode-10-ais-future-and-dangers-katja-grace.html), like a year ago. In other old news, I am opposed to the vibe of time-sensitivity often implicit in the public conversation.
# Inner alignment: what are we pointing at?

Proof that a model is an optimizer says very little about the model. I do not know what a research group is studying outer alignment is studying. Inner alignment seems to cover the entire problem at the limit. Whether an optimizer is  mesa or not depends on your point of view. These terms seem to be  [a](https://www.lesswrong.com/posts/HYERofGZE6j9Tuigi/inner-alignment-failures-which-are-actually-outer-alignment) [magnet](https://www.lesswrong.com/posts/SzecSPYxqRa5GCaSF/clarifying-inner-alignment-terminology) [for](https://www.lesswrong.com/posts/AHhCrJ2KpTjsCSwbt/inner-alignment-explain-like-i-m-12-edition) [confusion](https://www.lesswrong.com/posts/xdtNd8xCdzpgfnGme/clarifying-the-confusion-around-inner-alignment) [and](https://www.lesswrong.com/posts/HtEffpHcLxppLN6dL/explaining-inner-alignment-to-myself) [debate](https://www.lesswrong.com/posts/SzecSPYxqRa5GCaSF/clarifying-inner-alignment-terminology). I have to do background reading on someone to even understand what claim they're making. These are all indicators that we are using the wrong terms.

What are we actually pointing at? What questions do we want answered?

1. Do we care if a model is an optimizer? Is it important whether it is creating plans through an explicit search process or a clever collection of heuristics? A poor search algorithm cannot plan much and clever enough heuristics can take you to any goal. What's the important metric?
2. Sometimes a model will have great capacity to shape its environment but little inclination. How to divide between capacity and inclination in a way that closely corresponds to agents and models as we observe them? (One could say that capacity and inclination cannot be separated but the right definitions would split them right apart.)
3. When you specify what you want the model to do in code, what is the central difficulty? Is there a common risk or error in giving examples and giving loss/reward/value functions that we can name?
4. Is there a clear, accepted term for when models do not maintain desired behavior under distribution shift?
5. Should we distinguish between trained RL models that optimize and spontaneous agents that emerge in dynamical systems? One might expect the first to almost always happen and the second very rarely. What's the key difference?

I'll post my answers to these questions in a couple days but I'm curious how other people slice it. Does "inner alignment failure" mean anything or do we need to point more directly?
# Summaries: Alignment Fundamentals Curriculum

The linked document provides my summaries for most core readings and many further readings of [the alignment fundamentals curriculum composed by Richard Ngo](https://docs.google.com/document/d/1mTm_sT2YQx3mRXQD6J2xD2QJG1c3kHyvX8kQc_IQ0ns/edit), as accessed from July to early September 2022. Additionally, it often contains my preliminary opinions on the texts. Note that I’m not an expert on the topic. 

I have read all texts while simultaneously doing full-time work unrelated to AI alignment, and thus, due to time constraints, many summaries probably contain mistakes,  and my opinions would change upon further reflection. Additionally:

*   I only streamlined the process after a few weeks
*   the summaries of the first weeks are of lower quality, and more of them or my opinions are missing
*   Some summaries are also missing since I had a minor repetitive strain issue along the way, and since the curriculum changed while reading through it
*   Sometimes, the formatting is not ideal since I originally wrote the summaries on a slack channel and then copy-pasted them to google docs

Nevertheless, I was told that these summaries are useful, and therefore I’m sharing them with the wider community of people interested in alignment. 

If anyone wants to contribute their own summary, please put a suggestion into the google doc, and I will accept it with an attribution to the (optionally anonymous) author.

*Acknowledgments:* I want to thank Albert Garde, Benjamin Kolb, Fritz Dorn, Jens Brandt, and Tom Lieberum for discussions on the curriculum.
# Book Swap

**Summary:** Bring a book you liked, pitch it to the crowd, then trade your book for a book someone else brought that sounded good. This has no specific rationalist content.

**Tags:** Repeatable, medium

**Purpose:** Get people talking about books they read and encountering new ones. 

**Materials:** A whiteboard or piece of paper with a clipboard, plus appropriate writing implement. A timer. (Or a smartphone with a timer app.) 

**Announcement Text: **

Hello all!

I read a lot of books, which presents me with two problems. First, I constantly need recommendations for more books to read, and second, I constantly want to share books with other people so I'll be able to talk about the book with them. I would like to solicit your help with both of these problems. To that end, I’m hosting a book swap. The rules are simple. Bring a book. By the end of the night, leave with a book that isn't the one you brought.

Advice on choosing your book: The only two hard restrictions on your choice of book is that 1. it must be a physical book, the kind you could put into someone else's hand, and 2. it must be for an English speaking audience. Given my social groups I expect certain literary preferences will exist but, well, you're invited, and you like the books you like, so feel free to pick any book you like and it's quite plausible someone else has the same tastes as you.

We will have a whiteboard or similar writing surface. As you arrive, write your name and the title of your book on it. Then at the appointed time, we'll start running through the list- when it's your turn, you'll have two minutes by-the-clock to pitch why people should read your book. Then the swapping will begin in earnest! Please feel free to ask people questions about their books, talk about what kinds of books you like, or just hang out and socialize about anything else.

**Description:** 

As they arrive, make sure to collect people’s names and books. Stack the books in a spot where everyone can see them, in order if possible. Flip the stack (so the first person’s book is now on top) right before pitching the books. 

When it’s time to pitch books, be handy with a timer. You should be cheerfully ruthless about cutting people off once they run out of time, though two minutes tends to be enough that many people will finish with time to spare.

**Notes: **

This event mostly runs itself, which makes it a great pick if you’re a new organizer.

It’s marked as Repeatable, but you should pay attention to how fast people in your group tend to read books. The best cases are when they have time to read the book they took, and can bring that book back to pitch again. I would very roughly say that a month to two months is ideal.

The announcement assumes an English speaking audience. This event will work just fine in other languages, but it’s useful to pick a language you expect everyone to read fluently. If for some reason you don’t have a common lingua franca in your group, maybe pass on this.
# $13,000 of prizes for changing our mind about who to fund (Clearer Thinking Regrants Forecasting Tournament)

We have $13,000 of prizes you can win for our [Clearer Thinking](https://www.clearerthinking.org/) Regrants Forecasting Tournament on Manifold Markets! You can win money by:

(1) providing us with arguments or evidence that changes our minds about which of the 28 finalist projects to fund or how much to fund each project

(2) being one of the 20 most accurate forecasters at predicting which projects we end up selecting for funding in the Clearer Thinking Regrants program

You can learn more about the tournament (including all terms and conditions) here:

[https://manifold.markets/group/clearer-thinking-regrants/about](https://manifold.markets/group/clearer-thinking-regrants/about)

And you can begin forecasting here:

[https://manifold.markets/group/clearer-thinking-regrants/markets](https://manifold.markets/group/clearer-thinking-regrants/markets)
# Features and Antifeatures

One interesting thing I've noticed in the field of game design is that there are some purported features that are so hated that *not having them* becomes an advertised feature! Here are some quick examples:

*   In the world of card games, *Magic: the Gathering*'s 'lands'/'mana' system for accumulating resources is widely hated, and competitors to *Magic* at times explicitly advertise "no mana flood! no mana screw!" (For those unfamiliar with Magic, this is a situation where randomness in the game's resource system can essentially decide the game without the players making very meaningful decisions.)
*   In digital games, I've seen "no pay-to-win" used as a selling point to differentiate from games where players can directly buy things that increase their in-game power.
*   An even more extreme example of "no pay-to-win" is "no NFTs" -- NFTs have a very bad reputation in the gaming world, to the point where not having them is advertised sometimes even when one doesn't much have reason to think the game *would* have NFTs.

I like the word 'antifeatures' for these sorts of things -- and while the existence of 'antifeatures' first came to my attention in the domain of game design, I think this sort of thing is prevalent in a lot of other places as well. For example, some bars will advertise "no cover charge", some comedy shows will advertise "no drink minimum", and so on.

However, I think there's another important aspect to this that concept some people miss. *Just avoiding things that people don't like doesn't make your game good!* It's cool that your game has no mana screw or mana flood, that it isn't pay-to-win, and that it doesn't have NFTs -- but in order for me to be interested in it, it has to go beyond merely not having bad things and instead has to *actively have things to recommend it!*

In other words, I'm talking to my friends about a game and trying to get them to play it, I think I generally need more to go on than "it doesn't have <unpopular feature X>" -- there have to be other things that make it actively appealing!

One potentially relevant example is that there are a bunch of conservative films that have "this film isn't 'woke' like what Hollywood puts out!" or whatever as a primary selling point. Unfortunately, to be really successful you also have to... actually make a good film? "Not being <undesirable thing X>" seems generally insufficient.
# Can you define "utility" in utilitarianism without using words for specific human emotions?

I'm trying to get a slightly better grasp of utilitarianism as it is understood in rat/EA circles, and here's my biggest confusion at the moment.

How do you actually define "utility", not in the sense of how to compute it, but in the sense of specifying wtf are you even trying to compute? People talk about "welfare", "happiness" or "satisfaction", but those are intrinsically human concepts and most people seem to assume non-human agents at least in theory can have utility. So let's taboo those words, and all other words referring to specific human emotions (you can still use the word "human" or "emotion" itself if you have to). Caveats:

1.  Your definition should exclude things like AlphaZero or a $50 robot toy following a lights spot.
2.  If you use the word "sentient" or synonyms, provide at least some explanation of what do you mean by it.

If the answer is different for different flavors of utilitarianism, please clarify which one(s) your definition(s) apply to.

Alternatively, if "utility" is defined in human terms by design, can you explain what is the supposed process for mapping internal states of those non-human agents into human terms?
# Announcing AISIC 2022 - the AI Safety Israel Conference, October 19-20

ALTER, with the University of Haifa and the Technion, is excited to announce the details about its upcoming conference introducing AI safety in Israel. We’re going to be hosting Stuart Russell, as well as several other AI safety researchers, to speak about the field. We're also hosting a number of Israeli academics, who will speak about their related research.

The goal of the conference is to build a community of interest around AI safety in Israel, let people know about the field, and following up on the conference, we hope to connect interested attendees with work going on internationally. (This is not intended to be a conference where novel AI safety work is presented, and the primary audience is people who would otherwise be working on AI capabilities research.)  
  
If you are, or know anyone who is interested in AI safety in Israel, we would be happy for them to [register and attend](https://aisic2022.net.technion.ac.il/)!
# Tuesday Family Dinner

While I like Boston for [lots of reasons](https://www.jefftk.com/p/why-boston), one of the more personal ones is that my family is here. My dad and sister (and my other sister with her family until recently) are in the next town over, and my cousin's family is five towns NNW. For something like seven years now we've been getting together almost every Tuesday evening for family dinner.

[![](https://www.jefftk.com/tuesday-family-dinner-1.jpg)](https://www.jefftk.com/tuesday-family-dinner-1-big.jpg)

[![](https://www.jefftk.com/tuesday-family-dinner-2.jpg)](https://www.jefftk.com/tuesday-family-dinner-2-big.jpg)

My dad cooks, we eat together and talk. We all clean up after: the adults have all lived in my dad's house at some point so we all know where everything goes. The playroom is still well-stocked with toys even though kids don't live there anymore, and toys are more fun when you only see them once a week. It's very relaxed, and I'm really glad I can keep close to my folks and see them [so much](http://seeyourfolks.com/).

I feel very lucky that I happened to grow up in an area which also made sense for me to live as an adult, and I wish more people didn't have to choose between proximity to family and the other things they want in life.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02rvqGonXpV5cyWqwenUSLU1QLhVMXbsKxofBFVBtfoUBL5Jf97mGmqVe16sfhfraql)*
# What Do AI Safety Pitches Not Get About Your Field?

When I was first introduced to AI Safety, coming from a background studying psychology, I kept getting frustrated about the way people defined the and used the word "intelligence". They weren't able to address my questions about cultural intelligence, social evolution, and general intelligence in a way I found rigorous enough to be convincing. I felt like professionals couldn't answer what I considered to be basic and relevant questions about general intelligence, which meant that I took a lot longer to take AI Safety seriously than I otherwise would have. It feels possible to me that other people have run into AI Safety pitches and been turned off because of something similar -- a communication issue because both parties approached the conversation with very different background information. I'd love to try to minimize these occurrences, so if you've had anything similar happen, could you please share:   
  
**What is something that you feel AI Safety pitches usually don't seem to understand about your field/background? What's a common place where you feel you've become stuck in a conversation with AI Safety pitches? What question/information makes/made the conversation stop progressing and start circling? **  
  
(Cross-posted from  the EA forum)
# Making Prunes

Since [organizing a group flour buy](https://www.jefftk.com/p/organizing-a-group-buy-of-flour) in April 2020 I've been occasionally putting in orders at a [restaurant supply store](https://www.baldorfood.com). Generally this happens because we're running out of chocolate: I've failed at maintaining my [taste for bad chocolate](https://www.jefftk.com/p/cultivating-inexpensive-tastes), but if we buy in bulk we can get some [very good vegan chocolate](https://www.baldorfood.com/product/bakerypastry/spd2cc-semi-sweet-chocolate-chips) for $4.20/lb ($0.26/oz). A lot of what they sell is quite expensive, and everything else is in pretty large quantities, but lots of interesting options! This time I decided to get half a bushel of prune plums.

This is really quite a lot of plums: about 22lb. After eating all we wanted I was worried they might go bad soon, so the kids and I decided to try making prunes. We cut the plums in half, took out the pits, and put them on a rack on a baking sheet. I put in a pizza stone to keep the oven temperature steady, and set the oven to "warm":

[![](https://www.jefftk.com/prunes-in-oven-140F.jpg)](https://www.jefftk.com/prunes-in-oven-140F-big.jpg)

While the oven claims this is 170F, they pretty consistently read at 140F, which is a much better temperature for this anyway.

After a day or so they tasted about right, so I took them out of the oven. You can see that there are quite a few missing from sampling them as they cooked:

![](https://www.jefftk.com/prunes-out-of-the-oven.jpg)

The flavor got much more intense, beyond what you would expect from simple concentration.

[![](https://www.jefftk.com/finished-prune.jpg)](https://www.jefftk.com/finished-prune-big.jpg)

The prunes had lost some juice to dripping down into the pan. I scraped it up and it's good fruit leather:

[![](https://www.jefftk.com/prune-fruit-leather.jpg)](https://www.jefftk.com/prune-fruit-leather-big.jpg)

I liked these enough that I decided to do another batch of plumns, this time for a ["peach" cake](https://www.jefftk.com/recipes/peachcake). I didn't halve them or remove the pits this time, because they don't need to hold together, and I didn't use a rack:

[![](https://www.jefftk.com/plums-for-peach-cake.jpg)](https://www.jefftk.com/plums-for-peach-cake-big.jpg)

After about a day I took them out and removed the pits:

[![](https://www.jefftk.com/plums-for-peach-cake-removing-pits.jpg)](https://www.jefftk.com/plums-for-peach-cake-removing-pits-big.jpg)

I made the peach cake, which is essentially a half inch of cake on top of a half inch of cooked fruit, and brought it to Tuesday family dinner. While the kids weren't fans it was popular with the adults. Forgot to take a picture, sorry!

Now I'm cooking the rest of the plums down to keep in the fridge as something a bit jam-like (but not jam, [that's obsolete](https://www.jefftk.com/p/jam-is-obsolete)).
# A ranked link of LessWrong tags/concepts

*Epistemic status - a quick contribution. Possibly useless for anyone else. Also, a sort of feature suggestion.*

This post provides [a link](https://docs.google.com/spreadsheets/d/19aua5QRhH1YoqFpYRT3gVn_cq5lW_85vFICvZA621Jw/edit#gid=0) to a Google sheet which ranks tags/concepts discussed on LessWrong by frequency of mention. I am not sure if it will be useful, but it was for me, so I thought I'd quickly share it.

What did I do?
==============

On 22/9/22 I copied the tags from the [Concepts Portal](https://www.lesswrong.com/tags/all) into a Google sheet. I did some text splitting and concatenation to split out the frequency counts and produce a long table ranked by frequency. I couldn't copy over the links, but they are in the column A, which can be unhidden. Below are the top 20 rows. 

| **Concept** | **Mentions** |
| --- | --- |
| AI  | 3382 |
| World Modeling | 2353 |
| Rationality | 2278 |
| Practical | 1667 |
| World Optimization | 1527 |
| Community | 1110 |
| Covid-19 | 860 |
| Site Meta | 548 |
| Open Threads | 458 |
| AI Risk | 439 |
| Fiction | 414 |
| Politics | 305 |
| Book Reviews | 303 |
| Ethics & Morality | 293 |
| Decision Theory | 282 |
| Logic & Mathematics | 265 |
| Social & Cultural Dynamics | 256 |
| Economics | 245 |
| Scholarship & Learning | 245 |
| Forecasting & Prediction | 243 |

Why did I do this? 
===================

I am meeting more people in the rationalist community, and wanted a quick way to understand their key interests. I thought that seeing the frequency of tags/concepts used on LW would be a helpful overview. 

What next?
==========

If it seems useful for other people and is relatively easy, then it may be a good idea to allow users to rank tags by frequency of use on the [Concepts Portal](https://www.lesswrong.com/tags/all). 

Right now, I'd personally be interested in seeing a list/ranking of the newest and trending tags (e.g., those that have growing fastest in the last month/year) on the concept's portal page, but that could just be related to my perhaps unusual interest in understanding the focus of the rationality community. It might also be me projecting an unrealistic level of sustained interest going forward.
# UI/UX From the Dark Ages

(Inspired by a spirited discussion on #lesswrong)

You may not realize it, but our existing UI/UX paradigms are mainly from the 1970s, when human-oriented computing was still in dark ages. Menus, forms, check boxes, search bars, file systems, you name it. They are fine tools, polished by decades of doing the best we could with limited computing power we had, but they are still very much dated. Various attempts to introduce better UX have fallen flat due to limitations of algorithms, memory and compute. Bold attempts like [Microsoft Bob](https://en.wikipedia.org/wiki/Microsoft_Bob) and [Clippy](https://en.wikipedia.org/wiki/Office_Assistant) failed because they made user experience worse, not better. 

Things are different now. Natural language processing in limited domains is good enough or nearly good enough to provide an audio- and typed [DWIM](https://en.wikipedia.org/wiki/DWIM) interface without forcing the user to go through the levels of menus (usually poorly designed by the developers). This is not to replace the menus or the check boxes, but to add another way to do something. Here are some things you might say or type in an ever-present "do box" (not just a "search box"), some of which already sort of work on some gadgets:

*   "call mom on whatsapp" 
*   "play that song I used to listen to a lot last year"
*   "turn on the dark theme"
*   "show me a list of academic articles on ..."
*   "play the soup nazi Seinfeld episode on the big TV"
*   "send me an email when someone replies to my comment or post"
*   "connect wireless headphones I just turned on"
*   "connect to the public wifi nearby"
*   "mark me as being on vacation from now till the end of the month"
*   "open diary entry from ... or the nearest one with discussion of topic X"
*   "print a black and white copy of this document I am working on"
*   "insert a link to the wiki entry on UI/UX in this post"

Google Assistant, Siri and ilk can already recognize and interpret many of those, and execute some if asked just right, but it is still an afterthought most of the time, compared to the traditional way to do things. My point is that the technology is good enough now to make a natural language interface a convenient way to do things that require digging through menus and settings, and searching for solutions online. But the UI/UX mindset is still stuck in the old days, and it is basically hit-and-miss. 

This is a bit ironic, given that this UI/UX is front and center in sci-fi movies and shows, where, in addition to pressing some weird symbols on a colorful star trek touch panel, one can also say "Computer, find all Ferenghi ships nearby and list those that recently stopped at Risa" or "set course to Star Base 23" or "recalibrate tachyon sensors".

My hope is that the UI/UX paradigm will evolve to take advantage of the new AI capabilities and will require less memorization and digging through the forest of settings hidden in the bowels of each app or device. 

Of course, a next logical step is a version of Scott Alexander's [Magic Whispering Earring](https://archive.ph/30TA6), though maybe not as sinister.
# Framery Phone Booth CO2 Accumulation

The lab the NAO works out of has a [Framery O](https://www.frameryacoustics.com/en/products/framery-o/tech-specs/) phone booth, for taking calls. It's very good at sound isolation, but being so small (~1.5 m3, 54 ft3) you [might be worried](https://news.ycombinator.com/context?id=32928629) about CO2 building up:

[![](https://www.jefftk.com/framery-o-booth-door-open.jpg)](https://www.jefftk.com/framery-o-booth-door-open-big.jpg)

I decided to test it. Here are two rounds of sitting in the booth until the meter (Temptop M2000) shows CO2 levels plateauing, then exiting and leaving the door cracked until CO2 has returned to baseline:

[![](https://www.jefftk.com/framery-o-co2-accumulation.png)](https://www.jefftk.com/framery-o-co2-accumulation-big.png)

The booth is advertised as having 21.5 L/s (45 CFM) of mechanical ventilation, and you can see it works well. With the ventilation disabled, however, it reaches 2k PPM CO2 in 12min, 2.5k in 25min, and plateaus at ~2.7k.

Ours was unplugged, I think because someone assumed the ventilation sound was a white noise generator. If it was just for noise masking it wouldn't be needed: the sound isolation is very good. But since it's for ventilation we should be keeping it plugged in. It's still very quiet: I measure -25db on my Mac in Audacity with the mic volume all the way up vs -39dB with the ventilation disabled. Even with the ventilation running it's still slightly quieter than the office with no one talking, just light noise from the building HVAC (-22db). The only case I'd see unplugging it was if you were using the booth for sound recording, in which case I'd probably make a habit of opening the door to let it air out between takes.
# Double Asteroid Redirection Test succeeds

Existential risks are important, and asteroid impact is one kind of existential risks. Thankfully, compared to other existential risks, asteroid impact is well funded.

Double Asteroid Redirection Test launched in November 2021, and just (September 2022) impacted Dimorphos, a moon of an asteroid 65803 Didymos. This "double asteroid" system was chosen because we can precisely measure velocity change of eclipsing binary systems.

Let us congratulate NASA and humanity for the job well done!
# A Prince, a Pauper, Power, Panama

Instrumental convergence is an issue I think about a lot because it’s so natural. Being rich and powerful almost never works against you.

Take The Prince and the Pauper. It’s asymmetric.

The pauper’s side of the story can only happen because the prince allowed it, but the prince can just wear rags without anyone’s permission. It’s easy to go from prince to pauper, but not the other way around.

More abstractly, one with resources and power can always just leave it all behind and go live in the woods. Going the opposite way is The Story of Civilization.

What _is_ power? I think of it as a large action space, with large orbits in the state space induced by that. A fancy way of saying “it’s having a lot of options”. Hence, the idea that power is the ability to do what you want. An omnipotent one would be maximally powerful, because it can do _anything_. Largest possible action (and state) space.

So one would want to gain power, pretty much independent of its goals. Other universal instrumental goals are lots of mass-energy under your control and minimizing energy usage for a given level of function. Influence –in the literal sense of matter flowing towards you– and the Free Energy Principle.
# About Q Home

*(I didn't know how to name the post.)*

I have an internet friend in Ukraine. My love is in Belarus. I have a non-binary friend in Russia who is a former conscript. I'm a young male in Russia. I'm at risk, but might be relatively safe. 

Writing this post to highlight my main ideas (I would like to discuss them while I have the time). The ideas are about Alignment, subjective experience, and informal semantics.

*   *About Alignemt:*  [LW draft](https://www.lesswrong.com/posts/7zwrna5TPCCRp8eHP/a-single-principle-related-to-many-alignment-subproblems), [google doc](https://docs.google.com/document/d/1KB0hl-oLwziXT4Ktr77dxFq0uNSa1O075wYRtdgdqaQ/edit?tab=t.0) (A single principle related to many Alignment subproblems?).
*   *Ideas which got popular on Reddit:* [a holistic way to analyze fictional characters](https://np.reddit.com/r/CharacterRant/comments/uukfez/we_need_to_develop_new_ways_to_analyze_characters/), [chess positions and synesthesia](https://np.reddit.com/r/chess/comments/vh8494/chess_synesthesia_positions_are_like_paintings/), very short and weird music loops which evoke contradictory emotions ([version 1](https://np.reddit.com/r/LetsTalkMusic/comments/pib9d4/otherworldly_opening_chords/), [version 2](https://np.reddit.com/r/musictheory/comments/t3dj0l/music_superstimulus/)), [analyzing language with "possibilities"](https://np.reddit.com/r/DC_Cinematic/comments/tgblbi/the_batman_2022_speech_patterns_of_5_characters/).

If the opportunity cost is not too high for you, please help me to explain some of those concepts to you and other people.

...

Since **24.02.22** I desperately tried to share my ideas on Reddit (and later here) while simultaneously trying to advance them in case my posts won't be successful. 

I can't imagine what other people feel in this tragedy. I have a goal and it fills my mind and lets me remember the good things. I can't imagine how others live.

*Edited, shortened this post. If this date is not changed after **27.04.26**, then something happened to me. *

...

Whatever I'm writing about now, here's my main latest draft: [Google Doc](https://docs.google.com/document/d/1pwM2JHq6HZ4l5fG1cPS-ba2XfmjGalqWJecPPgxH_lo/edit?tab=t.k39kwudb1uph) (archetypes, recursive optimizers). A secondary draft: properties of reasoning ([LW draft](https://www.lesswrong.com/posts/AZmstnFhkdzPQ3hhN/topological-properties-of-reasoning)). Third: [Substack](https://thelostjockey.substack.com/p/surreal-jokes-draft) (using a weird informal concept to explore semantics).
# A Library and Tutorial for Factored Cognition with Language Models

We want to advance [process-based supervision for language models](https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes). To make it easier for others to contribute to that goal, we're sharing code for writing compositional language model programs, and a tutorial that explains how to get started:

*   The [Interactive Composition Explorer (ICE)](https://github.com/oughtinc/ice) is a library for writing and debugging compositional language model programs.
*   The [Factored Cognition Primer](https://primer.ought.org/) is a tutorial that explains using examples how to write such programs.

We've been using ICE as part of our work on [Elicit](https://elicit.org/) and have found it useful in practice.  
 

Interactive Composition Explorer (ICE)
--------------------------------------

[ICE](https://github.com/oughtinc/ice) is an open-source Python library for writing, debugging, and visualizing compositional language model programs. ICE makes it easy to:

1.  Run language model recipes in different modes: humans, human+LM, LM
2.  Inspect the execution traces in your browser for debugging
3.  Define and use new language model agents, e.g. chain-of-thought agents
4.  Run recipes quickly by parallelizing language model calls
5.  Reuse component recipes such as question-answering, ranking, and verification

ICE looks like this:

![](https://user-images.githubusercontent.com/382515/192681645-6ed87072-2dc6-4982-92d1-8de209bc3ef6.png)

Factored Cognition Primer
-------------------------

The [Factored Cognition Primer](https://primer.ought.org/) is a tutorial that explains (among other things) how to:

1.  Implement basic versions of amplification and debate using ICE
2.  Reason about long texts by combining search and generation
3.  Run decompositions quickly by parallelizing language model calls
4.  Use verification of answers and reasoning steps to improve responses

The Primer looks like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/73b3a075b17edef15187ddc0eb5333e8da4ab480cb8f904b.png)

  
If you end up using either, consider [joining our Slack](https://join.slack.com/t/ice-1mh7029/shared_invite/zt-1guya9qf7-y_N0Uv0nEt3vMTuaXp5k3Q). We think that factored cognition research parallelizes unusually well and would like to collaborate with others who are working on recipes for cognitive tasks.

To learn more about how we've been using ICE, watch [our recent Factored Cognition lab meeting](https://www.youtube.com/watch?v=cZqq4muY5_w).
# Should reasonably healthy people still take Paxlovid?

Should a non-immunocompromised, fairly healthy person in their late 30s take Paxlovid if they get one of the currently circulating strains of Omicron?

How would that answer change if:

*   Their symptoms are fairly mild (bad cold)
*   They have a high enough BMI to qualify for Paxlovid, but not a super high BMI?  
     

My previous understanding was that it was a good idea to reduce the severity of the case and risk of long-term effects.

However, a few friends' advice leads me to think that taking Paxlovid for a mild case does not make sense for a reasonably healthy non-elderly person:

*   "Paxlovid's usefulness is questionable and could lead to resistance. I would follow the meds and supplements suggested by FLCC [https://covid19criticalcare.com/.../i-care-early-covid.../"](https://covid19criticalcare.com/.../i-care-early-covid.../") \[not sure how reliable that site is, as it recommends Ivermectin, though it seems to be composed of actual MDs\]
*   \[3 anecdotes of nasty rebounds\]
*   Twitter threads suggesting Paxlovid reduces your protection against later reinfection \[which doesn't really seem like the most important thing to be optimizing for\]
    *   [https://twitter.com/michaelzlin/status/1574823881528864768?s=20&t=1W5akHDBMpAY-PboFqe3lA](https://twitter.com/michaelzlin/status/1574823881528864768?s=20&t=1W5akHDBMpAY-PboFqe3lA)
    *   [https://twitter.com/michaelzlin/status/1573835507842551809?s=20&t=1W5akHDBMpAY-PboFqe3lA](https://twitter.com/michaelzlin/status/1573835507842551809?s=20&t=1W5akHDBMpAY-PboFqe3lA)

On the other hand:

*   Another friend said: "Eric Topol twitter vaguely seems to suggest that data are broadly indicative of protection; haven't heard of much side effects"
*   (And 2 friends in their 30s took it and were fine)