Harley-ml/lesswrong huggingface dataset:
# Negativity enhances positivity

A pet peeve of mine has always been how people decide to give ratings on Yelp. If the restaurant is pretty good, it gets five stars. If it's ok, it gets four stars. If it's genuinely bad, it gets three. If the waiter was rude, it gets one.

This leads to a situation where most ratings range from 4 to 5. A 4.8 is a good rating. A 4.2 is underwhelming.

I think this is something that we've all internalized. In some sort of a priori sense, you'd think that an average of 4 out of 5 stars is pretty good. However, we all know that it isn't. In fact, this extends beyond Yelp. I remember hearing that Uber drivers get fired if their rating drops below 4.7 or something.

This works because deep down, we're all Bayesians. We don't take things literally. If we did take things literally, we'd see a place with an average of 4.2 stars and expect good things. After all, Yelp labels a 5 star review as "Great", 4 stars as "Good", 3 as "Ok", 2 as "Could've been better", and 1 as "Not good".

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bc419264d9f6727ba2582f58b0d1929ba305453320ea4f94.png)

Instead of taking things literally, we look at the rating as Bayesian evidence. We realize that a 4.2 rating isn't something that we'd expect to observe if the restaurant actually is really good, and it is something we'd expect to observe if the restaurant is mediocre.

So then, maybe there's no problem here. Maybe things all work out at the end of the day. The fact that raters lean so heavily towards giving ratings of 4 and 5 doesn't actually prevent users from using the average rating to tell how good a restaurant is. Users just need to re-calibrate.

It's the same thing as that friend who is overenthusiastic in their text messages. Everything has a bunch of exclamation points and emojis. And so, when you receive a text saying "good to hear from you", you're concerned. If they actually were happy to hear from you, you'd expect something more like "HEY!!! So good to hear from you!!!!!!!". If they were just normal-pleased to hear from you you'd expect "Hey! Great to hear from you!!!".

I think in most contexts this sort of skew towards positivity is fine. People are able to read well enough between the lines. Recalibrating would take more effort, require too much coordination, and cause too much social friction for it to be worth it. However, I can imagine contexts where accurate feedback is essential, and it would be worth the effort to incorporate 1 and 2 star reviews, so to speak.

Such contexts are probably rare. The participants would have to *really* care about being effective. Off the top of my head, a few candidates that come to mind are alignment researchers, cofounders of a startup, and a mentee working with a mentor. In situations like office jobs where people only *pretend* to care about being effective, pursuing such a culture would probably just backfire and cause friction.
# Two Percolation Puzzles

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d8ac38a892cc798055b355b90b2ba68c723dea60a77fe9a4.png)

Take a very large chessboard (NxN, where N is huge).

Remove some fraction **1-p** of the squares at random, leaving a fraction **p** of them.

Can you place a queen on the first row and then, via some sequence of legal moves, get it to the last row?

This is probabilistic, of course, but it turns out the probability of success undergoes a [sharp phase transition](https://en.wikipedia.org/wiki/Percolation_theory) \-\- near-zero for small values of **p**, then suddenly rising almost to one in the vicinity of a critical value **p_queen**.

For different pieces, the critical value is different, e.g. **p_rook** for a rook. (Note that **p_queen** <= **p_rook**.)

**Question 1:** What is **p\_queen + p\_rook**?

**Bonus Question:** Why didn't I ask you for the values **p_queen** and **p_rook** separately?

Now let's invent a new chess piece, the *bondsman*. This piece can move like a rook, and is also allowed to move along *northeast-southwest* diagonals between *white* squares, and along *northwest-southeast* diagonals between *black* squares. (There's also the *antibondsman* who has the same abilities, but with the words "white" and "black" swapped.)

**Question 2:** What is **p_bondsman**?

**Bonus Question:** Why did I call the piece that?

Answers to follow in a later post.
# Dominant Assurance Contract Experiment #2: Berkeley House Dinners

As an afterthought at the end of my [previous post](https://www.lesswrong.com/posts/9cvK8QX9ZRBxmEfcz/consider-paying-for-literature-or-book-reviews-using) on dominant assurance contracts, I decided on a whim as I was typing to append a small dominant assurance contract:

> In theory, writers could kickstart posts using dominant assurance contracts. An example (this is a real offer): If you send $20 to arjun.panickssery at Gmail via PayPal by noon New York time on January 21st, I'll send you back $25 if fewer than 10 people sent me money. If 10 or more people send me money, I'll post a review of Steven Pinker's The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century by the end of the month. I'm not sure whether I'm just giving away free money right now.

I received exactly nine out of the ten funders needed and lost $45.

Now a second attempt, this time in the real world: at [Andromeda House](https://twitter.com/andromeda_house) we plan to host large weekly dinners on Monday evenings for the local EA/rationality/etc community at our house in Southside Berkeley.

Here’s how it works:

- You can [Venmo me](https://venmo.com/?txn=pay&audience=public&recipients=Arjun-Panickssery&amount=20&note=dinner) (@Arjun-Panickssery if the link doesn't work) or [PayPal me](https://paypal.me/arjunpanickssery) any amount of at least $20 with the subject line "dinner" or similar. 
- If I get at least $700 total by noon Pacific time on July 15, I'll host dinners from July 17 till the end of August (seven dinners). 
- If I get less than $700 total, I'll give you a 25% return (e.g., if you sent me $100, I'll send you back $125).

**Example**: If you live in town and this service would be worth more than $20 per dinner to you and you'd expect to come to three before September, you should pay at least $60, since you either get a service worth more than that amount or you make a 25% return. You could try to free-ride but the fact that my previous contract received just one less than the target number suggests that I'm well-calibrated.

If you don't live in town—or even if you do—you can also idly speculate if you think that I'm unlikely to hit my target and likely to pay out.

You could also just pseudo-donate money without regard to whether you think I'm calibrated just because you think that DACs are cool and should proliferate.
# Final Lightspeed Grants coworking/office hours before the application deadline

This is the last day to apply to [Lightspeed Grants](https://www.lesswrong.com/posts/xQ4ajnzavSgbYiko2/launching-lightspeed-grants-apply-by-july-6th)! 

It's also the last office hours during which you can co-work with others on your application and ask me and Sydney questions about the process. 

> Is the application *really* only 2 hours though?
> ------------------------------------------------
> 
> Often, applicants get nervous about grant applications and spend a lot more time than they need to on them, or get overwhelmed and procrastinate on applying.
> 
> We really just want you to spell out some basic information about your project in a plain way and think this is doable in the 1-2 hour timeframe.
> 
> If you're worried about overthinking things, we'll have application co-working sessions and office hours every Thursday of June (*edit: and July 6th*) between noon and 2PM PT. If you think you might procrastinate on the application or get stuck in the weeds and spend a ton of unnecessary time on it, you can join one and fill out the application on the call, plus ask questions. [Add the co-working to your calendar here](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MWFqYTdzc3A5ajU4bHVwbnVsbXBzdTQyY29fMjAyMzA2MDhUMTkwMDAwWiAzMTEzNTRhOWNiNmYzNGY3MTkxYzQ2YWM1ODViM2FjZWJmZTgzNjczNTJhYTI3YTcyZWEzNjEyZTY4Nzc1YTlkQGc&tmsrc=311354a9cb6f34f7191c46ac585b3acebfe8367352aa27a72ea3612e68775a9d%40group.calendar.google.com&scp=ALL), or join the Google meet directly at that time [here](https://meet.google.com/dhw-hzve-cyh)!
# Puffer-pope reality check

*This post is humor. It's meant to be entertaining and nothing more.                                          **tl;dr:** nothing quite summarizes the weirdness of the 21rst century as well as this picture.*

I was sorting through my mind dump recently and found a "first-principles rant" on the widely circulated image of the pope in a puffer jacket. 

![](https://lh4.googleusercontent.com/DuS_usEFdRLG-XxU7jCitNeVVtWlU4s8AOm_wx77MhJbBj9DfkP9yrXGaPe6lq_OvGObm2hoZAtKxJi1QveYKBX8RvgKGMQpSXBgxpZlHXKFHSszyLMwU2pXem4X5Hdyq4Fo9KTxHJ2lYCmtOsu5JGo)

This image was: 

1.  Built using technology that effectively amounts to magic.
2.  Magic controlled by a guy on shrooms. The talking space monkeys that we are have developed magic, only to use it to make silly-looking pictures as we chemically disrupt our brains for the fun of it. 
3.  Was generated thousands of kilometers away from where its high creator came up with the prompt. (For free.)
4.  The idea behind computers is essentially: "what if we put lightning through a maze? If the maze is built in just the right way we can make the lightning *do math,* because that's how geometry works apparently."
5.  This image was generated blindly through multiplying matrices. It is purely the result of calculations, nothing more. 
6.  While it took a ton of fine-tuning to get here, we frankly have no idea how Midjourney works. Fine: by changing the geometry of the universe we are capable of making it generate puffer-pope pictures for us; but we're not sure how it works exactly because we're training our models at the macro-level (and the micro-level is really really complex, and not as efficient as it could be, but that doesn't matter to us because we have a lot of spare compute). 
7.  The Pope is a figure that most people on Earth know. Why? Because we have this thing called religion. Religion is a meme which relies on believing--without evidence-- that the universe is essentially sentient and came up with us on purpose. 
8.  Religion is really viral apparently. The drippy guy in the pic is the spiritual leader ("") of over *1,300* *million* humans, distributed all over the planet.
9.  Human language is oddly efficient. The Midjourney prompt was a single sentence long: "Pope Francis in a puffer jacket walking through the streets of Rome".[^2g1fquubeu1]  This reminds me that you can also get incredibly specific results with just 5 words or so on Google. 
10.  I've seen this picture and you've seen this picture and somehow this picture has become famous on a global scale. We're really damn interconnected aren't we? 

[^2g1fquubeu1]: Upon further research, I discovered that the original prompt may have had extra directions, like what filter to apply and the right lighting conditions. That doesn't change this point much, however. In just a few dozen words instead of 12, this incredibly specific image was dragged out of the entropic depths of the universe's white noise.
# [Linkpost] Introducing Superalignment

Some quotes:

> **Our approach**
> ----------------
> 
> Our goal is to build a roughly human-level [automated alignment researcher](https://openai.com/blog/our-approach-to-alignment-research). We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.
> 
> To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:
> 
> 1.  To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to [assist evaluation of other AI systems](https://openai.com/research/critiques) (*scalable oversight).* In addition, we want to understand and control how our models generalize our oversight to tasks we can’t supervise (*generalization)*.
> 2.  To validate the alignment of our systems, we [automate search for problematic behavior](https://www.deepmind.com/blog/red-teaming-language-models-with-language-models) (*robustness)* and problematic internals ([*automated interpretability*](https://openai.com/research/language-models-can-explain-neurons-in-language-models)).
> 3.  Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (*adversarial testing*).
> 
> We expect our research priorities will evolve substantially as we learn more about the problem and we’ll likely add entirely new research areas. We are planning to share more on our roadmap in the future.

\[...\]

> While this is an incredibly ambitious goal and we’re not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:[^C^](https://openai.com/blog/introducing-superalignment#fn-C)
> 
> There are many ideas that have shown promise in preliminary experiments, we have increasingly useful metrics for progress, and we can use today’s models to study many of these problems empirically. 
> 
> Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company.
# AI Intermediation

Why is Google such a valuable company? More than half of its money comes from search ads: when people are looking to spend money it's very common that they use Google. Even slightly influencing spending decisions, with the scale of their userbase, lets them bring in massive amounts of money. They are a very successful intermediary.

Right now I bet there are dozens of startups trying to figure out how they can build AI assistants to be even better intermediaries. The business plan looks something like:

*   Find some area where you can put in a bit of work and get an AI assistant that is seriously useful to some group of people. So useful that they love it and get their friends to start using it. Maybe you start with people trying to buy houses who get a ton of mostly useless emails with new listings but also the occasional gem. It doesn't matter whether there's a commercial tie-in at this stage, though, you just want to be great.
    
*   Keep expanding the areas your AI assistant is useful in. Talk to your most enthusiastic users, find related areas you can save time and hassle, grow, grow, grow.
    
*   Some of these areas will have natural commercial applications; make deals where you send them a bunch of traffic in exchange for a small discount, and keep part of the discount. Disclose this to your users so the FTC doesn't get mad at you!
    
*   Stay focused and execute better than your competitors. It's a crowded space, but also one with a lot of potential.
    
*   Eventually you have a large number of people where you're buying things on their behalf and exercising substantial discretion in doing so. They're happy with your purchases, and your position as a trusted intermediary lets you bring in a ton of money.
    

I haven't yet met an AI assistant I'd trust to act on my behalf, and prompt injection is pretty concerning. But as companies figure out scalable oversight and improve reliability I expect we'll get there.

(Prompted by helping a friend think through a business plan for a startup, though with how crowded the space is they weren't able to get funding and decided to do something else.)
# Open Thread - July 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](http://lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](http://lesswrong.com/s/yai5mppkuCHPQmzpN).
# What are the best non-LW places to read on alignment progress?

I've lately been thinking I should prioritize a bit more keeping up with alignment-relevant progress outside of LessWrong/Alignment Forum. 

I'm curious if people have recommendations that stand out as reliably valuable, and/or have tips for finding "the good stuff" on places where the signal/noise ratio isn't very good. (Seems fine to also apply this to LW/AF)

Some places I've looked into somewhat (though not made major habits around so far) include:

*   Blogs of OpenAI/Deepmind/Anthropic
*   reddit.com/r/ControlProblem 
*   Stampy Discord 
*   EleutherAI Discord

I generally struggle with figuring out how much to keep up with stuff – it seems like there's more than one full-time-job's worth of stuff to keep up with, and it's potentially overanchoring to think about "the stuff people have worked on" as opposed to "stuff that hasn't been worked on yet."

I'm personally coming at this from a lens of "understand the field well enough to think about how to make useful infrastructural advances", but I'm interested in hearing thoughts about various ways people keep-up-with-stuff and how they gain value from it.
# ask me about technology

I have a lot of opinions about technology - what's feasible, what isn't, what the best ways to do things are, and so on. Sometimes I write them down on my [blog](https://www.bhauth.com/blog/), but usually I don't. I also know some people who are good at designing technology.

If you want my personal opinion on some technology or technological problem, I guess you can ask me here.
# Can LessWrong provide me with something I find obviously highly useful to my own practical life?

When I meet new people, I often like to figure out how I can be useful in a really short amount of time, just by talking. 

Example:   
Often this will involve evangelizing some items from a short list of technologies that have impacted my life a lot, such as erotic hypnosis (you can use this to have really good sex over any distance or even asynchronously, those 2 facts alone have a lot of lifechanging implications imo).

Telling people about the (very large and real) efficacy of erotic hypnosis is useful because most people go from not imagining this is possible to imagining its possible and having a clear path to test it out in a very short amount of time. If it works, the benefit is obvious, and I give a lot of update about whether it works + info about how to see for themselves quickly and easily. 

Another example:  
In 2021, Sapphire gave me really good intel about founding crypto companies and really good generic arguments about why the EMH is false and therefore I should in fact act upon the good intel.

Another example:  
A post about how candles are bad for you

Non-example:  
When I try to extract useful conversations from others, a common reply is "if you meditate a lot its really good". That's not useless to hear, I guess (I don't meditate and wouldn't know), but it's not super useful since people say it all the time and it's not really that easy to verify for myself. 

I would really like a forum where the entire point of the forum was stuff like this: Read this -> instantly or near-instantly become way smarter about things of actual practical relevance to your life. To me, this is the best possible kind of post in the world, and the space of possible posts of this kind is so big that you could just fill a forum with only that. Why bother with anything else? 

As far as I can tell, LessWrong is not such a forum, but people do try to do this some of the time successfully. I really want to engage with the people who are trying to do this, but I'm lazy and I don't really know how to easily find them. 

So I'm wondering if people who are more familiar with LessWrong can/would easily provide me with content that is immediately obviously highly useful to me.
# Introducing bayescalc.io

I made a simple online calculator for doing elementary hypothesis testing!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/df1954653cbb62f749e1123570b758373318caa72232036f.png)

[Link to example shown.](https://bayescalc.io/#KCdoeXBvdGhlc2VzLSdIZWFsdGh5J35IYXMgZGlzZWFzZUFwKjkuMCwxLjA0cG9zdGUqMS43Nzc3Nzc3LDguNTU1NTU1NTI0ZXZpZGVuY2UtJ1Rlc3RlZCBwb3NpdGl2ZSBmb3JBbGlrZWxpaG9vZHMtWzAuMSwwLjldXSkqcmlvcl9vZGRzLTktIVs0XX41MzM3NjZBIFgnNAFBNzU0LSpf)

I was disappointed that an intuitive and easy-to-use app for using bayes' theorem apparently did not exist, so I decided to make it. My goal was to make something that:

1.  Helped people correctly and quickly evaluate the effect of evidence while comparing hypotheses.
2.  Was easy enough for someone who didn't know math to use.
    1.  And which also helped *show* what the math was doing in an intuitive way, so that you didn't have to trust math you didn't understand.
3.  Felt good enough to use that it would actually be used.
4.  Could be used to share simple models of things in a way that would help people have more productive discussions, and promote shared model building.

Hopefully I've at least made substantial progress on these goals, and I'd really appreciate feedback on ways in which it falls short! This includes even minor interface or design issues. You can leave feedback as a comment here, or on the [issues page](https://github.com/adelelopez/bayes/issues).  
  
I'd also be really happy to see people share examples they've made in the comments!
# Writing this post as rationality case study

\[Before I begin: If you don't like this post, please let me know why. Even just few words like boring/off-topic/poorly-written may give me something to work with.\]

I wish to share a late struggle I have with rationality, because I think that it touches some interesting points. But more importantly - because I think that it is important to think about rationality in the context of concrete day-to-day decisions. This post is going to be messy and have no specific "point" or definite conclusion - like real life decisions.

Like many, I am attracted to content creation. I like to think about stuff and share my insights and patterns of thoughts, and like the idea that one day I will be able to spread them far and wide. To begin moving in that direction, I lately published two posts. The first was ignored, and the second heavily downvoted. Then I stopped to think.

Why do I want that really? The immediate flattering answer is that I value good ideas and view their creation as the highest public service. The less flattering answer is that I like prestige and want more of it. The interesting answer is that the will to create content is not that different from the will to have children: The memes that ended up inhabiting my brain are those who successfully spread themselves across society - often by convincing their host that meme-spreading is a good idea. It mean that I should expect exactly what I see - that the will to spread memes will be wide-spread, and therefore highly competitive. That people put extraordinary effort into writing books with very small success rates, when the positive black swans are just not big enough to justify it (if money was the sole consideration).

I first thought about it as a bias for content-creation, and wanted to write a post about that, but I probably don't endorse this framing anymore. Let's say that I buy that analysis - does it really mean that I shouldn't pursue that goal? It seem less respectable now, for sure. It seem less likely to maximize my happiness or my contribution to society, but it doesn't make me not want it. Is it a terminal value of mine, or just a tool that I refuse to put down when I see it don't work? How would I tell? Does the distinction even make sense given that I'm not an ideal agent?

When did all those thoughts come up anyway? When I saw the downvotes and wanted to shy away. Downvotes which I pretty much expected, and which should therefore not have cause a significant update. Motivated reasoning my way out of difficult situation. So... what? What should I do when I find an argument convincing, but know that it was generated by flawed motivations?

Now I think those where all the wrong questions, or maybe the right questions in the wrong time - big, and abstract, and all inside my brain. It will make sense to ask whether my goal is worth extraordinary effort, and how extraordinary exactly it would have to be - **after** I make some ordinary effort. No point to worry about it after I was merely trying to try, and found out that trying is hard...
# Announcing AI Alignment workshop at the ALIFE 2023 conference

The upcoming [ALIFE 2023 conference](https://2023.alife.org/) is hosting a workshop on AI Alignment and Artificial Life. This will complement the [special session](https://humanvaluesandartificialagency.com/) on AI Alignment at the conference. The workshop will include presentations and a discussion panel from established AI Alignment and ALife researchers, exploring the overlap between these two fields.

**Please see the workshop website for more information: **[https://humanvaluesandartificialagency.com/workshop/](https://humanvaluesandartificialagency.com/workshop/) 

### **Logistics**

**Date**: Friday 28th July, 2023

**Venue**: Hybrid: Sapporo Japan and remote

Attending remotely requires paying the [remote registration fee](https://2023.alife.org/registration/). The workshop organisers are offering a limited number of bursaries for anyone who cannot afford the registration fee. See the [registration page](https://humanvaluesandartificialagency.com/registration/) on the workshop website for details.  
 

### **What is Artificial Life?**

Artificial Life (ALife) is a field which studies the properties of life and how the processes of living systems can be recreated synthetically.

Many of the ideas and concepts from AI Alignment overlap with those in ALife. In particular, questions about autonomy, agency and goal-directedness in artificial systems.

  
Please don't hesitate to contact me (Rory) with any questions.
# H5N1. Just how bad is the situation?

How bad do you think the [H5N1](https://forum.effectivealtruism.org/topics/h5n1-virus) situation is? I tried to search on LessWrong and EffectiveAltruism, but haven't found that much discussion.

When I was reading about **H5N1**, a lot of my assumptions were broken. I was shocked how vulnerable we are to the most "predictable" threats.

Should/can rationalists take this threat into account? Can this threat warrant even, I dunno, some community-wide preparation?

**Edit:** [this comment](https://www.lesswrong.com/posts/2wyz8yQEevJF8GDGo/h5n1-just-how-bad-is-the-situation?commentId=YHPkuGznKHfKWks6L) gives specific predictions. Can somebody explain the justification behind such predictions? What exactly keeps the risk relatively low?
# Whether LLMs "understand" anything is mostly a terminological dispute

*Epistemic status: speculating on what other people mean when they make certain claims, without having done any checking to validate.  Probably (mostly) true anyways.*

Thought prompted by [Tsvi's latest post](https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce): most discussions of whether LLMs "understand" things seem to conflate a few claims under the term "understand":

1.  the weights encode an algorithmic representation of the "thing" in question
2.  the weights encode some kind of much more general reasoning ability, which is sufficiently capable that it can model a large number of not-obviously-related abstractions, including the one in question
3.  #2, but in a way which also includes some kind of map-territory distinction
4.  #2, but in a way which resembles human "system 2" thinking, where it's capable of doing the laborious explicit reasoning people sometimes do when other options aren't good enough

There are probably other things meant by the claim that I haven't considered.

I suspect that some people who say that LLMs don't understand things are thinking of 3 or 4, maybe 2.  1 seems pretty obviously true (we have numerous examples: [modular arithmetic](https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-anal), [Othello](https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world), [mazes](https://www.lesswrong.com/posts/cAC4AXiNC5ig6jQnc/understanding-and-controlling-a-maze-solving-net), etc).  I wouldn't bet against 2; evidence in favor includes some speculation[^4g7saofe9n5] that one reason the GPT-series of models are so much better than everything else is the substantial amount of code in their training data[^l8e81j1tz79].  I don't think 3 or 4 have happened yet.

Once one's gotten past the [linguistic and philosophical confusions](https://www.lesswrong.com/posts/WBdvyyHLdxZSAMmoz/taboo-your-words), such arguments could in principle be resolved with empiricism.  In practice, most such arguments probably aren't worth the time.

[^4g7saofe9n5]: I don't remember where exactly I saw this claim.  If anybody has a source I'd appreciate it. 

[^l8e81j1tz79]: The causal chain being something like "code is more structured than natural language, and that structure includes a lot of very powerful, generalizable insights, useful across many domains".  I'm not sure if this was spelled out in the source or if that was me filling in the blanks.
# What to read on the "informal multi-world model"?

People on LessWrong often talk in terms of an "informal many-world model". For example they talk about the worlds where the alignment problem is relatively easy vs the worlds where the alignment problem is really hard. 

I wonder what are good references for this "informal multiverse model of reality" with multiverses of worlds with different properties? What's the history of this line of thoughts.

(And I also wonder about possible mathematical models of that, how much people are talking about those, and how much people keeping some of those mathematical models in mind? I've seen some sheaf-based models which seemed to be like that.)
# Why have exposure notification apps been (mostly) discontinued?

I recently noticed that exposure notifications for COVID-19 in the app that I had been using, called CA Notify, had stopped working (I had attended an event a few days prior, and was feeling ever-so-slightly unwell, so I thought to check to see if there were any exposure notifications from the event; anyway it looks like my ever-so-slight not-feeling-well was unrelated to the event and it self-recovered quickly).

After digging further, I found that most of the COVID-19 contact tracing apps stopped sending exposure notifications around May 11, 2023, when the COVID-19 pandemic was officially declared over. See e.g. https://www.cnbc.com/2023/05/12/covid-most-americans-wont-receive-exposure-alerts-on-phones.html

I think this is a bit ... sad? I feel like contact tracing / exposure notifications is one of the kinds of precautionary measures that provides benefits at ~0 personal cost, unlike things like social distancing, masking, and vaccinations. In fact I would have liked exposure notifications to be expanded beyond just COVID-19 to cover more respiratory diseases and also include the ability to report new ailments _before_ they spread widely. Seems like keeping an exposure notification system going and having people regularly use it to report infections would be good practice for both known and unknown ailments.

I hope that companies and health institutions are secretly working on some more robust longer-term solution such as this, but have not seen any evidence of it so far.

What are the reasons this might have been shut down? I couldn't find much discussion online, so I tried to come up with my own ideas, and here they are; I am not personally convinced of any of them:

1) Maybe the cost of running the backends was very high for the institutions managing the app (the California government in case of CA Notify).

2) Maybe people think "ignorance is bliss" when it comes to getting notified of COVID exposures, insofar as after having received such notifications, they may be morally obliged to cancel meetings with other people, incurring financial and social costs of doing so. They'd rather *not* be notified and incur the increased risk of transmitting COVID.

3) Maybe the exposure notification system was considered a security risk or liability risk of some sort, so shutting it down was considered prudent as a way to reduce the risk of some major scandal. (I think, even if this is the case, it's probably better to tweak and fix any such issues now than try to rush out another iteration during the next pandemic!)

4) Maybe there's evidence/data that the exposure notifications system didn't work at all? Or that *not* receiving an exposure notification gave people false confidence that they don't have COVID?

NOTE: Originally posted to Facebook at https://www.facebook.com/vipulnaik.r/posts/pfbid02Q1jFK5LB7KhGWhj2rEteud8QfFrB2UxhyZYbWxQzSZWrpigJSasJEX7mVZKYmBZPl but I didn't get responses on Facebook and I think LessWrong might be a better venue for discussing this.
# Consider Joining the UK Foundation Model Taskforce

A few months ago, Ian Hogarth wrote the Financial Times Op-Ed headlined “[We must slow down the race to God-like AI](https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2).”

A few weeks ago, he was appointed head of the UK Foundation Model Taskforce, and given 100 million pounds to dedicate to AI safety, [to](https://twitter.com/NPCollapse/status/1670374034767216641) [universal](https://twitter.com/Simeon_Cps/status/1670396708587487232) [acclaim](https://twitter.com/robertwiblin/status/1671123009187246081). Soon there will [also be a UK Global AI Summit](https://twitter.com/JgaltTweets/status/1669627294124593153).

[He wrote an op-ed in The Times asking everyone for their help](https://www.thetimes.co.uk/article/how-ill-help-make-the-ai-revolution-safe-mj0zx00k6), [with accompanying Twitter thread](https://twitter.com/soundboy/status/1670343527723679744). Based on a combination of sources, I am confident that this effort has strong backing for the time being although that is always fragile, and that it is aimed squarely at the real target of extinction risk from AI, with a strong understanding of what it would mean to have an impact on that.

Once again: The real work begins now.

The UK Taskforce will need many things in order to succeed. It will face opposition within and outside the government, and internationally. There is a narrow window until the AI summit to hit the ground running and establish capability and credibility.

The taskforce represents a startup government mindset that makes me optimistic, and that seems like the best hope for making government get things done again, including on other vital causes that are not AI, and not only in the UK. We likely only get one shot at this. If the taskforce fails, there will probably not be another such effort.

Right now, the main bottleneck is that the taskforce is talent constrained. There is an urgent need to scale up rapidly with people who can hit the ground running and allow the taskforce to orient.

If you are in position to help, then with the possible exception of creating your own organization at scale, I believe this is the highest leverage opportunity currently available.

To reach out and see if you can help, [you can fill out this Google Form here](https://docs.google.com/forms/d/e/1FAIpQLSeOV3FPChEstFy7CcuHc6af1Jv50H0jfxi7dkieyj5gHabsiw/viewform).
# ACX Montreal Meetup July 22nd 2023

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup! This week, we're keeping it open-ended for a freewheeling discussion.

**Venue**: L'Esplanade Tranquille, 1442 Clark, indoors. When you have reached the second floor, go to the right.

**Date & Time:** Saturday, July 22nd, 2023, 1:00-4:00 PM.

Please RSVP! RSVPing is not mandatory but it helps with planning the event and encourages others to come.

Email me at [acxmontreal@gmail.com](mailto:acxmontreal@gmail.com) if you have any questions.

Use [this form](https://forms.gle/CUqzSzAxJ4yfJtpL6) if you want to be added to the ACX Montreal mailing list.
# Two Hot Takes about Quine

I read Quine's *Word and Object* on vacation last week. Overall it was fine, but there were two things that might be worth quick mentions.

Quine, Supervised Learning Supremacist
--------------------------------------

One important facet of the book is Quine's picture of how humans learn language. It's not *quite* that Quine is a behaviorist, though the influence is overt. Fortunately I live in 2023 when we have the abstractions of supervised and unsupervised learning, because that's a much better-fitting category to stick Quine's picture in than "behaviorism."

Quine's picture of how humans learned things was all about supervised learning. The baby learns to say "mama" because the parents reward speaking behavior similar to it. Sure, there's some unsupervised learning that has to happen somewhere, but Quine basically sweeps it under the rug to get back to talking about supervised learning.

This is foundational to the book. Later arguments that trade in notions like similarity of concepts or simplicity of concepts are all relying on how concepts would be learned if we just used supervised learning. Which is a problem because (of course) humans don't actually work that way. A more accurate picture of human learning would have big impacts on the conclusions of the book, ranging from adding more subtlety to the whole radical translation question to exposing a bunch of claims about ontology as being based on preconceptions.

For the time, Quine was taking advantage of advances in theoretical sciences to say new and interesting things in philosophy. It's not that he was being dumb. This tale is really about how, in the intervening 60 years, progress in neuroscience, psychology, mathematics, and computer science has sneakily also been progress in philosophy.

Literal language of thought
---------------------------

Quine uses a model of cognition based on language. GOFAI before AI. Insofar as reasoning is logical, Quine treats it as consisting of logical rules (not necessarily deductive) acting on a cognitive state that corresponds to a sentence.

Sometimes caveats are raised, but then later forgotten/elided. Quine seems to say that maybe the language of thought model is only *approximately* correct, but it's close enough to be getting on with. Of course, that isn't true, and we should ditch the language of thought.

This is another one of those things that's both informative about the book's historical context, and would make more sense if I knew more of the history of the ideas here. Though I'm not motivated enough to get more of the historical picture by reading Carnap et al.

My headcanon is that the language of thought picture is motivating for the middle of the book, where Quine lays out a systematization of the rules of language, and then explores some ways one could make them even simpler and more regular. He associates simple, regular rules for language with a simple, regular core of cognition. If instead we modeled thought as having intermediate states that don't map neatly onto language, linguistics becomes more of a descriptive endeavor, and less of a royal road to the mind.
# Announcing The Roots of Progress Blog-Building Intensive

Today we’re launching [The Roots of Progress Blog-Building Intensive](https://fellowship.rootsofprogress.org/), an 8-week program for aspiring progress writers to start or grow a blog.

**What:** Through this program you will launch (or re-launch) a blog/Substack, get into a regular writing habit, improve your writing, and begin building your audience.

You will meet and learn from progress studies experts, authors, and think tank leaders; and join a community of peers. You’ll participate in a special cohort of Write of Passage, the online writing school, to create writing habits and develop a writing system–we’re partnered with them to tailor the instruction for deeply researched, long-form essays. At the end, we’ll all get together in person for a 3-day event in San Francisco.

**Why:** To [keep progress going](https://rootsofprogress.org/progress-studies-a-moral-imperative)—in science, technology, and industry—we have to [believe that it is is possible and desirable](https://rootsofprogress.org/a-new-philosophy-of-progress).

Today we lack a bold, ambitious vision for the future. It’s time for a new generation of intellectuals to help the world understand and appreciate progress.

**Advisors:** We have a fantastic lineup of advisors including Tyler Cowen (Mercatus Center), Allison Duettmann (Foresight Institute), Bret Kugelmass (Last Energy), Johan Norberg (author, *Progress: Ten Reasons to Look Forward to the Future*), Steven Pinker (author, *Enlightenment Now*), Virginia Postrel (author, *The Future and Its Enemies*), Alec Stapp (Institute for Progress), Derek Thompson (The Atlantic), Alex Trembath (Breakthrough Institute), Marian Tupy (HumanProgress.org), and Tamara Winter (Stripe Press).

**Who:** This program may be for you if you’re excited about progress studies and you love to write.

Maybe you’d like to explore a career in writing about progress, or maybe you’re already blogging but would like to get to the next level—find your own topic area, increase your productivity, get more plugged into the community, and grow your audience.

**Commitment:** 10–15 hours a week, for 8 weeks.

You’ll use the time to read, to write, to participate in discussions with experts, to provide editing and feedback to your peers, and to participate in group meetings.

There is no cost to you.

**When:** Program runs Sep 15–Nov 17. [Applications are open now](https://fellowship.rootsofprogress.org/), deadline Aug 11.

Special thanks to O’Shaughnessy Ventures for helping to sponsor this program!
# Do we automatically accept propositions? 

*Epistemic Status: Exploratory (I'm searching for an expert or someone who has studied and rigorously investigated these claims/essays, as I intend to, but time doesn't permit me now)*  
  
16 years ago, Eliezer asked "[Do we Believe Everything we're told?](https://www.lesswrong.com/posts/TiDGXt3WrQwtCdDj3/do-we-believe-everything-we-re-told)"  
  
This was with respect to the Spinozan Model of the mind.

> Spinoza suggested that we first *passively accept a proposition in the course of comprehending it*, and only afterward *actively disbelieve* propositions which are rejected by consideration. 

My question I'd like to pose is, has anyone deeply investigated this? Cognitive Science is of interest to me, but I am not as educated as many in this field.  
  
I'd like to call attention to work done by [Eric Mandelbaum](https://www.ericmandelbaum.com/about-me) of CUNY[^2czn0p23xc3].

Mandelbaum has investigated the Spinozan Model further and written multiple essays on it.

I want to reach out to the lesswrong community to see if anyone has deeply investigated the claim and research behind this paper [*Thinking is Believing*](https://philpapers.org/rec/MANTIB-2)*,* published in 2014. [^0pznk89c2w7]  
  
This idea of the Spinozan model has been on my mind for months, and upon taking an Intro to Philosophy of Cognitive Science class, I've wanted to dive into it more.   
  
What's stopping me is lack of trust in my ability, and other projects going on.   
I've been debating asking, but I figured I would put this out there to the community and hope I hear back.

  
**The importance:**  
IF we automatically accept propositions AND we are truth seeking, THEN we ought to guard our mind from likely falsehoods.

[^2czn0p23xc3]: Mandelbaum was a researcher at FHI in the past (date not found) 

[^0pznk89c2w7]: ABSTRACT The idea that people can entertain propositions without believing them is widespread, intuitive, and most probably false. The main goal of this essay is to argue against the claim that people can entertain a proposition without believing it. Evidence is presented demonstrating that we cannot withhold assent from any proposition we happen to consider. A model of belief fixation is then sketched and used to explain hitherto disparate, recalcitrant and somewhat mysterious psychological phenomena. The proposed model is one where beliefs are the automatic output of a computationally null belief acquisition reflex. In short, the model holds that the mere activation of a mentally represented truth apt proposition leads to immediately believing it. The essay concludes by considering some consequences that the proposed model of belief acquisition has for our concept of rationality.