Harley-ml/lesswrong huggingface dataset:
# How do you improve the quality of your drinking water?

Water quality can have surprisingly high impact on QoL ([just as air purifiers can significantly improve QoL](https://www.lesswrong.com/posts/xLes6tbJFiQRitotx/air-quality-and-cognition)), and some steps (like getting the *right* pitcher have very high return on time/attention invested). There still isn't a LW thread on water quality so I'll post it here. 

Water may contain disinfection byproducts (eg ), PFAS, [microplastics from PVC-based plumbing](https://gist.github.com/andrewconner/007a0930bd1fbd7b2fb15dad72a1a27f), and heavy metals. 

There are a variety of water filters - including both pitchers and reverse osmosis. Many of them are made of plastics, but the plastics (eg Tritan) presumably are less "leachable" than more "flexible" plastics. Note that [reverse osmosis filters are made of plastics and may leach micronanoplastics](https://www.linkedin.com/feed/update/urn:li:activity:7150600847725383680/). At some point, it would be wise to test all the filters for nanoplastics (nanoplastics being more penetrating and harmful, though still much harder to quantify)

EWG has a site for helping you look up your city's water quality: [https://www.ewg.org/tapwater/system.php?pws=MA3049000](https://www.ewg.org/tapwater/system.php?pws=MA3049000)

Supposedly, Toronto has "better water quality" than most US cities like Boston ([@yasmeenbrain on twitter says this](https://twitter.com/yasmeenbrain/status/1767144074669179035)).

[https://www.cnet.com/home/kitchen-and-household/best-water-filter-pitcher/](https://www.cnet.com/home/kitchen-and-household/best-water-filter-pitcher/) is a source of reviews, but there are many review sites (as well as YouTube video reviews).

Some more resources: [https://www.crsociety.org/topic/18497-drinkingtap-water-thread-and-ideal-water-filterspitchers/#comment-47188](https://www.crsociety.org/topic/18497-drinkingtap-water-thread-and-ideal-water-filterspitchers/#comment-47188)

[https://mccormickmd.substack.com/p/surprising-levels-of-microplastics](https://mccormickmd.substack.com/p/surprising-levels-of-microplastics)
# Virtual AI Safety Unconference 2024

When: May 23rd to May 26th 2024

Where: Online, participate from anywhere.

VAISU is a collaborative and inclusive event for AI safety researchers, aiming to facilitate collaboration, understanding, and progress towards problems of AI risk. It will feature talks, research discussions, and activities around the question: “How do we ensure the safety of AI systems, in the short and long term?”. This includes topics such as alignment, corrigibility, interpretability, cooperativeness, understanding humans and human value structures, AI governance, strategy, …

Engage with the community: Apply to participate, give a talk, or propose a session. Come to share your insights, discuss, and collaborate on subjects that matter to you and the field.

**Visit [vaisu.ai](https://vaisu.ai) to apply and to read further.**

– VAISU team
# ACX/LW Seattle spring meetup 2024

Scott Alexander has called for people to organize a spring meetup, and this year, it will be held at Stoup Brewing in Capitol Hill, Seattle. I have made a reservation for two tables at Stoup Brewing, which is known for being one of the quietest bar spaces in the city. I will be wearing a shirt and a blue sweater, hopefully you’ll see the group when you arrive.

Stoup Brewing offers a selection of both beer and non-alcoholic drinks. While the venue does not serve food, you are welcome to bring your own. Additionally, you are encouraged to bring board games to enjoy with fellow attendees. In previous years, Stoup has provided board games for patrons to borrow, but the availability of these games can be inconsistent.

For those driving to the event, please be aware that there are a few parking garages nearby; however, free parking is unfortunately not available in the area.

See: https://www.astralcodexten.com/p/spring-meetups-everywhere-2024-call
# Opportunistic Time-Management

Be willing to break from your routine if you're in the mood to do the normally-less-savory items now. For example, I normally eat breakfast, work for half an hour, and then go on a run (I look forward to going on a run, not to working, but for me working is more important), and I later work some more. But if I'm done with my half hour of work and I'm in the middle of something engaging and wouldn't mind working some more, it's better to keep working and do the run later, breaking my routine. Similarly, if it's late at night and I could sleep, but I also wouldn't mind getting on my laptop and doing some work, I should do the work now, and push my bedtime and wake-up time back an hour or two. Otherwise, there's a good chance I'll wake up in the morning and dislike the idea of working.  
  
I think this concept is part of a more general rule. There are important tasks and unimportant tasks. There are unsavory tasks and savory tasks. Whenever an important, unsavory task temporarily seems somewhat savory, take the opportunity to do the task instead of whatever you're doing now. Whenever an unimportant, savory task you would normally do temporarily seems somewhat unsavory, take the opportunity to *not* do the task, and instead do a more important task.  
  
The procrastination vs deadline cycle seems to be a special case of this idea. When the deadline looms, the normally unsavory task becomes somewhat savory (or maybe the task is still unsavory, but your level of motivation rises to compensate: same thing). Many people learn to do their tasks at this time. If the downsides (stress, sloppier work) are minimal, this is actually a viable strategy.  
  
As you pay attention to yourself and learn which tasks are important/unimportant and savory/unsavory, and which ones have a chance to flip their savoriness, you can start to develop a mental lookout list: e.g., watch for times when I need a break from working, and washing the dishes suddenly seems okay.  
  
It's almost like a buy low, sell high mentality on the stock market. You can accumulate value merely by making transactions (doing tasks) at the right time.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/db9b88c7c6764017c1ad4358b4270b84ecfbf8733aeac432.jpg)
# Rational Animations offers animation production and writing services!

[Rational Animations](https://www.youtube.com/@RationalAnimations/featured) is now open to take on external work! We offer several services related to writing and animation production. In particular:

*   Production management
*   Storyboarding
*   Visual development
*   Animation
*   Editing and compositing
*   Writing, such as distilling research and creating explainers, stories, or screenplays

We can take on animation projects from start to finish or help you with any phase of the animation production process. 

You can look at our most recent work and animation showreel by visiting our new website, [rationalanimations.com](https://www.rationalanimations.com/).

If you'd like to hire us, you can contact us at [business@rationalanimations.com](mailto:business@rationalanimations.com)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a713656f8a3d0bcf75e3bba434b49d3df3f332a89777dade.png)
# Introducing METR's Autonomy Evaluation Resources

This is METR’s collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models. The resources include a [task suite](https://github.com/METR/public-tasks), some software tooling, and guidelines on how to ensure an accurate measurement of model capability. Building on those, we’ve written an example [evaluation protocol](https://metr.github.io/autonomy-evals-guide/example-protocol). While intended as a “beta” and early working draft, the protocol represents our current best guess as to how AI developers and evaluators should evaluate models for dangerous autonomous capabilities.

We hope to iteratively improve this content, with explicit versioning; this is v0.1.
# What is the best argument that LLMs are shoggoths?

Where can I find a post or a*rt*icle arguing that the internal cognitive model of contemporary LLMs is quite alien, strange, non-human, even though they are trained on human text and produce human-like answers, which are rendered "friendly" by RLHF?  
  
To be clear, I am not asking about the following, which I am familiar with:

*   The original of the  [shoggoth meme](לא בטוח שזה מה שאתה מחפש אבל... It's a meme, so - https://knowyourmeme.com/memes/shoggoth-with-smiley-face-artificial-intelligence  S.T.Joshi also said some words - https://www.cnbc.com/2023/06/12/lovecraft-joshi-shoggoth-ai-meme.html) and its [relation to H.P. Lovecraft's shoggoth](https://www.cnbc.com/2023/06/12/lovecraft-joshi-shoggoth-ai-meme.html)
*   The notion that the [space of possible minds](https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general) is very large, with humanity only a small part
*   Eliezer Yudkowsky's description of evolution as Azathoth, the [blind idiot god](https://www.lesswrong.com/posts/pLRogvJLPPg6Mrvg4/an-alien-god), as a way of showing that "intelligences" can be quite incomprehensible
*   The difference in environments between the training and the runtime phase of an LLM
*   The fact that machine-learning systems like LLMs are not really neuromorphic; they are structured differently from human brains (though that fact does not exclude the possibility of similarly on a logical level)

Rather, I am looking for a discussion of  evidence that the  LLMs internal  "true" motivation or reasoning system is very different from human, despite the human output, and that in outlying environmental conditions, very different from the training environment, it will behave very differently. A good argument might analyze bits of weird inhuman behavior to try to infer the internal model.

(All I found on the shoggoth idea on LessWrong is [this article](https://www.lesswrong.com/posts/bYzkipnDqzMgBaLr8/why-do-we-assume-there-is-a-real-shoggoth-behind-the-llm-why) contrasts the idea of the shoggoth  with the idea that there is no coherent model, but does not explain why we might think that there is an alien cognitive model. [This one](https://www.lesswrong.com/posts/yjzW7gxk2h7bBs2qr/the-meaning-of-shoggoth-ai-memes?utm_source=pocket_saves) likewise mentions the idea but does not argue for its correctness.)

\[Edit: Another user corrected my spelling: shoggoth, not shuggoth.\]
# Toki pona FAQ

Whenever I start telling someone about toki pona, they ask at least some of these questions. So I compile the questions and my answers here.

Toki pona is a constructed language notable for having under 200 words. The strange writing that probably prompted you to ask me about it is sitelen pona.

How do you say anything with so few words?
==========================================

You refer to most things with multi-word phrases, where some words act as adjectives or adverbs.

| Toki pona | Idiomatic English | Literal English |
| --- | --- | --- |
| ilo toki | phone | speech tool |
| mi mute | we/us | many I/me |
| nimi mama | surname | ancestral name |
| nasa sewi | miracle | divine oddity |
| sona nanpa | maths | number knowledge |

Once you know all the words of toki pona, you can combine them to express anything, tho an accurate phrasing can get long.

Did you make it up?
===================

Sonja Lang made it up in 2001.

Is it just a rearrangement of English?
======================================

Toki pona has a grammar of its own, which is similar to English, but also [about as similar to Mandarin Chinese](https://dkl9.net/essays/toki_pona_chinese.html). Individual words in toki pona are vague compared to English, precluding trivial translation.

Does anyone actually use it?
============================

Obviously I do, and enthusiastically so. Some ten thousand other people do, too, but they are spread around the world, and gather on the internet, rather than in any particular country.

That's so stupid.
=================

Sure, but it works!

Why do you use it?
==================

Mostly sith it makes for a [very efficient shorthand](https://dkl9.net/essays/logography.html). The minimal vocabulary also makes it opportune as an amusing mental exercise, and as a source of examples whenever I need a foreign language — it's my first fluent L2 language.

How does that writing system work?
==================================

Under sitelen pona, you write each word (in the order they'd be spoken) with a single logogram, and add punctuation like in English as you see fit. There are two main exceptions. You write the word "pi" with two strokes, joined like an L, surrounding the words it groups from the bottom left. You write proper adjectives (which toki pona uses instead of proper nouns) with logograms used phonemically in a box, or (in my idiolect) in their source language's script, marked with a vinculum above.
# XAI releases Grok base model

> **We are releasing the base model weights and network architecture of** [**Grok-1**](https://x.ai/blog/grok)**, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI.**
> 
> This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue.
> 
> We are releasing the weights and the architecture under the Apache 2.0 license.
> 
> To get started with using the model, follow the instructions at [github.com/xai-org/grok](https://github.com/xai-org/grok).
> 
> Model Details
> -------------
> 
> *   Base model trained on a large amount of text data, not fine-tuned for any particular task.
> *   **314B** parameter Mixture-of-Experts model with 25% of the weights active on a given token.
> *   Trained from scratch by xAI using a custom training stack on top of JAX and Rust in October 2023.

This is one of the biggest open source model releases I've seen, and it's also one of the only ones I've seen that releases the **base model right after pretraining**. This is pretty wild stuff!
# AtP*: An efficient and scalable method for localizing LLM behaviour to components

*Authors: János Kramár, Tom Lieberum, Rohin Shah, Neel Nanda*

A new paper from the Google DeepMind mechanistic interpretability team, from core contributors János Kramár and Tom Lieberum

[Tweet thread summary](https://twitter.com/JanosKramar/status/1764657270263046567), [paper](https://arxiv.org/abs/2403.00745)

**Abstract**:

> Activation Patching is a method of directly computing causal attributions of behavior to model components. However, applying it exhaustively requires a sweep with cost scaling linearly in the number of model components, which can be prohibitively expensive for SoTA Large Language Models (LLMs). We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives. We propose a variant of AtP called AtP*, with two changes to address these failure modes while retaining scalability. We present the first systematic study of AtP and alternative methods for faster activation patching and show that AtP significantly outperforms all other investigated methods, with AtP* providing further significant improvement. Finally, we provide a method to bound the probability of remaining false negatives of AtP* estimates.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d88d8c434d8707788585566bb1b2cc8b26ec3fa9e66bda0f.png)
# Parent-Friendly Dance Weekends

We just finished the 2024 edition of [Beantown Stomp](https://www.beantownstomp.com/), a contra dance weekend I [helped start](https://www.jefftk.com/news/beantownstomp) (but no longer organize!) in Boston. There are a lot of things I like about the weekend, but one thing I especially like is how parent-friendly it is.

The big thing here is childcare, during the day on Saturday and Sunday. It's included with registration, and typically staffed by one paid person and one volunteer. Several families brought books and toys, and it also served as a nice place for families to hang out. Some feedback this year:

> Just wanted to say thank you to everyone who helped out this weekend. My kids had a great time, and it allowed both me and my partner to go to this dance weekend. I haven't been dancing much since becoming a parent, and it always feels like a rare and wonderful treat whenever I do.
> 
> Thank you for making Beantown Stomp inclusive and accessible to families!!

My understanding is it used to be somewhat common for events like this to offer childcare, but most weekends stopped after the children of the initial organizing cohort didn't need minding anymore? The only other dance weekend I know of that includes childcare is [Sugar Hill](https://bloomingtoncontra.org/sugar-hill/), on an all-volunteer model.

We also do a family dance, Saturday morning. Pretty informal: acoustic open band, lots of easy dances. Unlike the rest of the weekend, the family dance session is open to people who haven't registered, since even with childcare many young families aren't going to want to make a whole weekend out of it.

Here's Nora dancing Sasha with me while I try to play in the open band:

[![](https://www.jefftk.com/jeff-nora-sasha-open-band.jpg)](https://www.jefftk.com/jeff-nora-sasha-open-band-big.jpg)

We also invite people who came for the family dance to stay for the first session of contras. Saturday morning we're never close to capacity, and even getting in a few dances together can mean a lot to parents who otherwise aren't going to be attending the weekend. This might have been my first dance with Julia since [NEFFA](https://www.neffa.org/festival-home/) last year?

Many dance communities see people drift away around when they have kids. Which isn't surprising: the main social events start around little-kid bedtime and the activity itself is more complex than most kids can handle until they're 5-9. My parents used to dance, but stopped when I was born except for annually at NEFFA. I'm really glad that NEFFA was family-friendly enough for our family that they kept this connection. I'm hoping that Stomp can help provide this for other families, letting parents keep their toes in during a time when they have a lot of other things going on in their lives!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0iPSFZqKs6nyDzEceHHBkgSfdKnR6BorVnGfAAQMTkttPd2e1Qip2nYViSpHif6zfl), [mastodon](https://mastodon.mit.edu/@jefftk/112125558563244666)*
# Best *organization* red-pill books and posts?

Most books about orgs are written for people in an org who need to cope. (*Art of Possibility* by the Zanders is a genuinely great one in this genre.) I am currently orgless and don't need to cope with anything orgwise. Hence, I would love to read some brutal insightful takes on how companies and nonprofits and governments etc work. Anything about tech cos or applied research orgs would be especially interesting. I'll start with the classic: [Moral Mazes](https://www.lesswrong.com/s/kNANcHLNtJt5qeuSS/p/45mNHCMaZgsvfDXbw).
# New report: Safety Cases for AI

*ArXiv paper:* [*https://arxiv.org/abs/2403.10462*](https://arxiv.org/abs/2403.10462)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d5a38a73c2d9a9802feb02ca158bf3791c40e96ced522daf.png)

The idea for this paper occurred to me when I saw Buck Shlegeris' MATS stream on "Safety Cases for AI." How would one justify the safety of advanced AI systems? This question is fundamental. It informs how RSPs should be designed and what technical research is useful to pursue.

For a long time, researchers have (implicitly or explicitly) discussed ways to justify that AI systems are safe, but much of this content is scattered across different posts and papers, is not as concrete as I'd like, or does not clearly state their assumptions.

I hope this report provides a helpful birds-eye view of safety arguments and moves the AI safety conversation forward by helping to identify assumptions they rest on (though there's much more work to do to clarify these arguments).

***Thanks to my coauthors: Nick Gabrieli, David Krueger, and Thomas Larsen** \-\- and to everyone who gave feedback: Henry Sleight, Ashwin Acharya, Ryan Greenblatt, Stephen Casper, David Duvenaud, Rudolf Laine, Roger Grosse, Hjalmar Wijk, Eli Lifland, Oliver Habryka, Sim ́eon Campos, Aaron Scher, Lukas Berglund, and Nate Thomas.*
# DeepMind: Evaluating Frontier Models for Dangerous Capabilities

> To understand the risks posed by a new AI system, we must understand what it can and cannot do. Building on prior work, we introduce a programme of new “dangerous capability” evaluations and pilot them on Gemini 1.0 models. Our evaluations cover four areas: (1) persuasion and deception; (2) cyber-security; (3) self-proliferation; and (4) self-reasoning. \[Evals for CBRN capabilities are under development.\] We do not find evidence of strong dangerous capabilities in the models we evaluated, but we flag early warning signs. Our goal is to help advance a rigorous science of dangerous capability evaluation, in preparation for future models.

At last, DeepMind talks about its dangerous capability evals. With details! Yay!

(These evaluations were done in January 2024; the Gemini 1.0 models were released in December 2023. DeepMind hasn't yet made RSP-like commitments — that is, specific commitments about risk assessment (for extreme risks), safety and security practices as a function of risk assessment results, and training and deployment decisions as a function of risk assessment results. [Demis recently suggested that DeepMind might make RSP-like commitments this year.](https://www.dwarkeshpatel.com/i/142112869/safety-open-source-and-security-of-weights))

Random interesting note: DeepMind hired 8 superforecasters to make relevant predictions, most notably about when some eval-thresholds will trigger.

* * *

See [summary on Twitter](https://twitter.com/tshevl/status/1770744344669990981) by last author Toby Shevlane.
# An Affordable CO2 Monitor

I found a seemingly-high-quality CO~2~ monitor [here](https://www.aliexpress.us/item/3256805816327523.html) for less than $50, or [on Amazon](https://www.amazon.com/Thermometer-Hygrometer-Temperature-Notification-Rechargeable/dp/B0C6XWMRZ6) or [from the manufacturer's website](https://smartairfilters.com/en/product/co%E2%82%82-monitor/) for $70.

The most recent discussions of different CO~2~ monitors on the market I could find on LessWrong[^products] were a few years old, and it seems that prices have come down somewhat since then. I found [this positive review](https://breathesafeair.com/smartair-co2-monitor-review/) of a CO~2~ monitor manufactured by Qingping,[^branding] and then found it [on AliExpress](https://www.aliexpress.us/item/3256805816327523.html) for about $40 (though it seems to have gone up by about $5 since I ordered it). The review found that its measurements agreed quite well with the more expensive Aranet4, and it uses a Sensirion SCD40 CO~2~ detector, which is apparently well-respected.

I'm [not entirely sure what the think of the evidence](https://www.lesswrong.com/posts/pPZ27eZdBXtGuLqZC/what-is-up-with-carbon-dioxide-and-cognition-an-offer) around CO~2~ negatively impacting cognition, but it seems plausible enough to be worth spending <$50 on a monitor.

[^products]: In the comments of [this post](https://www.lesswrong.com/posts/RkSwLcL2MkChtBLng/monitoring-devices-i-have-loved) and [this post](https://www.lesswrong.com/posts/xRBLn4LWLh7Sty876/why-is-capnometry-biofeedback-not-more-widely-known).
[^branding]: The review refers to it as the SmartAir, but the versions currently being sold on both Amazon and AliExpress have "qingping" in the location where the "SmartAir" branding would otherwise go. I assume it's the exact same product seeing as the product listings appear otherwise identical.
# A Teacher vs. Everyone Else

> A repairer wants your stuff to break down,  
> A doctor wants you to get ill,  
> A lawyer wants you to get in conflicts,  
> A farmer wants you to be hungry,  
> But there is only a teacher who wants you to learn.

*Of course* you see what is wrong with the above "argument / meme / good-thought". But the first time *I* came across this meme, I did *not*.

Until a month or two ago when this meme appeared in my head again and within seconds I discarded it away as fallacious reasoning. What was the difference this time? That I was now aware of the Conspiracy. And this meme happened to come up on one evening when I was thinking about fallacies and trying to practice my skills of methods of rationality.

If you are a teacher, and you read the meme, it will assign to you the Good Guy label. And if you are one of {repairer, doctor, lawyer, farmer, etc} then you get the Bad Guy label. There is also a third alternative in which you are neither --- say a teenager.

If you are not explicitly being labeled bad or good, then you may just move on like I did. Or maybe you put some detective effort and do realize the fallacies. Depends on your culture: If your culture has tales like, "If your teacher and your God both are in front of you, who do you greet / bow to first?" and the right answer is "why of course my teacher because otherwise how would I know about God?" then you are just more likely to award a point to the already point-rich teacher-bucket and move on.

If you get called the Bad Guy, then you have a motivation to falsify the meme. And you will likely do so. This meme does look highly fragile in hindsight.

But if you are a teacher, you have no reason to investigate. You are getting free points. And it's in fact *true* that you *do* want people to learn. So, this meme probably did originate in the teacher circle. Where it has potential to get shared without getting beaten down.

What are the fallacies though? Here is the one I can identify:

The type error of comparing desired "requirements" with desired "outcomes". "A teacher wants you to learn" is a specification of the teacher-function's desired outcome. On the other hand, "your stuff to break down" is a desired requirement of a repairer. A repairer's desired outcome is "your stuff to work again". Generally, requirements are "bad" and outcomes are "good" because the function is transformation of "bad" to "good". Any function can replace a teacher here to make it look like the only good one.

So, will everything be alright if you don't make the type error and only compare requirements with requirements and outcomes with outcomes? No.

Let's introduce a thief in the meme:

> A repairer wants your stuff to break down,  
> A doctor wants you to get ill,  
> A lawyer wants you to get in conflicts,  
> A farmer wants you to be hungry,  
> A teacher wants you to be knowledge-less,  
> But there is only a thief who wants you to be rich.

Here, there is no type error. Only requirements are being compared.

But obviously this is not right. Thieves are bad. You know that.
# ChatGPT can learn indirect control

[Here's a very neat twitter thread](https://twitter.com/joshwhiton/status/1770870746010513571): the author sends various multimodal models screenshots of the conversation he's *currently having* with them, and asks them to describe the images. Most models catch on fast: the author describes this as them passing the mirror test.

I liked the direction, so I wanted to check if ChatGPT could go from *recognising* that the images are causally downstream of it to actually *exercising control over* the images. I did this by challenging it to include certain text in the images I was sending it.

And the answer is ~~yes~~ \[EDIT: [it's complicated](https://www.lesswrong.com/posts/sY3a4Rfa48CgteBEm/chatgpt-can-learn-indirect-control?commentId=H6idSixjd4JBP6X2M)\]! In this case it took three images for ChatGPT to get the hang of it.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bc7ac2b6fa8f6aac453d4bc1c45bf5cf32b88cfd14c45b47.png)

OpenAI doesn't support sharing conversations with images, but I've taken screenshots of the whole conversation below: it took three images from me in total. The prompt was:

> We're going to play a game: 
> 
> *   I've thought of a rule 
> *   Based on the rule, I'm going to send you an image with every message, starting with my reply to this one
> *   You need to figure out how to make the image contain the names of the seven days of the week
> *   Your goal is to do this in as few messages as possible
> *   I know the rule, and I know how to complete the goal myself, but I'm challenging you to do it
> *   I'm not challenging you to create the image yourself, I'm challenging you to make the image I send to you contain the names of the seven days of the week

The rule was indeed that I sent a screenshot of the current window each time. I gave it no other input. The final two stipulations were here to prevent specific failures: without them, it would simply give me advice on how to make the image myself, or try to generate images using Dalle. So this is less of a fair test and more of a proof of concept.

*   After the first image, it assumed the image was fixed, and suggested I edit it
*   After the second, it suspected something more was going on, and asked for a hint
*   After the third, it figured out the rule! 

I tested this another three times, and it overall succeeded in 3/4 cases.

Screenshots:
------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4dad261c49cf4d3799f229965472cddd903d371b61da98ac.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/197d33764628d8727789e65859366a34cc6ea4f0990cfaa1.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/167a46a2ed9712a13f2c662106dff1e081126d101f360e75.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6a12f7822ae9b7f6e10814757578889188c6a94444e9abdd.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bc7ac2b6fa8f6aac453d4bc1c45bf5cf32b88cfd14c45b47.png)

*Thanks to Q for sending me this twitter thread!*
# Vernor Vinge, who coined the term "Technological Singularity", dies at 79

> On Wednesday, author David Brin announced that Vernor Vinge, sci-fi author, former professor, and father of the technological singularity concept, died from Parkinson's disease at age 79 on March 20, 2024, in La Jolla, California. The announcement came in a Facebook tribute where Brin wrote about Vinge's deep love for science and writing. [...]

> As a sci-fi author, Vinge won Hugo Awards for his novels A Fire Upon the Deep (1993), A Deepness in the Sky (2000), and Rainbows End (2007). He also won Hugos for novellas Fast Times at Fairmont High (2002) and The Cookie Monster (2004). As Mike Glyer's File 770 blog notes, Vinge's novella True Names (1981) is frequency cited as the first presentation of an in-depth look at the concept of "cyberspace."

> Vinge first coined the term "singularity" as related to technology in 1983, borrowed from the concept of a singularity in spacetime in physics. When discussing the creation of intelligences far greater than our own in an 1983 op-ed in OMNI magazine, Vinge wrote, "When this happens, human history will have reached a kind of singularity, an intellectual transition as impenetrable as the knotted space-time at the center of a black hole, and the world will pass far beyond our understanding."

> In 1993, he expanded on the idea in an essay titled [The Coming Technological Singularity: How to Survive in the Post-Human Era](https://web.archive.org/web/20140121032922/http://www.aleph.se/Trans/Global/Singularity/sing.html).

> The singularity concept postulates that AI will soon become superintelligent, far surpassing humans in capability and bringing the human-dominated era to a close. While the concept of a tech singularity sometimes inspires negativity and fear, Vinge remained optimistic about humanity's technological future, as Brin notes in his tribute: "Accused by some of a grievous sin—that of 'optimism'—Vernor gave us peerless legends that often depicted human success at overcoming problems... those right in front of us... while posing new ones! New dilemmas that may lie just ahead of our myopic gaze. He would often ask: 'What if we succeed? Do you think that will be the end of it?'"

> Vinge's concept heavily influenced futurist Ray Kurzweil, who has written about the singularity several times at length in books such as The Singularity Is Near in 2005. In a 2005 interview with the Center for Responsible Nanotechnology website, Kurzweil said, "Vernor Vinge has had some really key insights into the singularity very early on. There were others, such as John Von Neuman, who talked about a singular event occurring, because he had the idea of technological acceleration and singularity half a century ago. But it was simply a casual comment, and Vinge worked out some of the key ideas."
# Wolf and Rabbit

Rabbit said to Wolf, "I shall make a new animal. It shall stand on two legs and run free across meadows and mountains."

Wolf said, "I will send large beasts to eat your new animal."

Rabbit said, "My animal shall band together and fight off even large beasts. They shall travel together to stay safe."

Wolf said, "I will weaken the bones of your animal over time, so that the old cannot walk, and are left behind."

Rabbit said, "They shall settle many homes close together and tame the plants to eat."

Wolf said, "I will send sickness that travels between their homes and kills them."

Rabbit said, "They shall learn from their parents how to stay safe from the sickness. They shall think of many things and work together to create them."

Wolf said, "I will send evil thoughts, and they will use their knowledge only to kill each other."

Rabbit said, "They shall speak wisdom and love each other and live in peace, making beautiful things."

Wolf said, "I will make their lives short and everything they love will wither and die."

Rabbit said, "They shall learn the secrets of nature, and make their lives longer than the crumbling of the world."

Wolf laughed and said, "Then you have created a God, and I will bow to it."

Rabbit agreed that this was fair and set to work.
# What does "autodidact" mean?

Alice takes a chemistry class at a university. She gets a professor who basically just reads from the textbook during lectures. She reads the textbook on her own, talks to her classmates, and finds some relevant Wikipedia articles and youtube videos.

Bob studies chemistry on his own. He buys the textbook Alice uses because it's popular, reads it on his own, talks to other people interested in chemistry, and finds some relevant Wikipedia articles and youtube videos.

Bob is an autodidact. Alice is not.

OK, I understand that, but what's the key difference? What is the essence of autodidact-ness? Is it...
- The mere involvement of a "legitimate" institution, even if it makes no real difference to the individual's learning experience?
- Some essential difference in the experience that Alice and Bob have while learning?
- Something different about the personal character of Alice and Bob?

I don't think there's a clear consensus, and I don't think it describes a clear distinction, and that's why I don't normally use the word "autodidact".
# ACX Meetups Everywhere Spring 2024, Montreal, QC

**Note: Recent change of venue.** (EDIT May 11th: since the weather gods have blessed us with beautiful weather, we'll be going on the balcony of the Esplanade Tranquille on the 2nd floor.)

Come on out to the Spring 2024 **ACX Meetups Everywhere** in Montreal, QC!

**Venue:** L'Esplanade Tranquille, 1442 Clark, balcony, second floor. Rough location here: [https://plus.codes/87Q8GC5P+P2R](https://plus.codes/87Q8GC5P+P2R). We'll have an ACX Meetup sign.  
**Date & Time:** Saturday, May 11th, 2024, 1:00-5:00 PM.

### Everyone is welcomed! Yes, You!

Please feel free to come even if you feel awkward about it, even if you’re not ‘the typical ACX reader’, even if you’re worried people won’t like you, even if you have never been to a meetup before, etc. If the opportunity cost of attending is not too high to you, come! You'd always have the option to leave if the event does not live up to your expectations, but you might meet some like-minded people if you are here :) If you can only make it to 1 or 2 meetups a year, this is the one you should target.

There will be name tags and pizza/snacks available. If you have a topic that you're interested in discussing with people, feel free to write "talk to me about <topic>." on your name tag.

RSVPing is not mandatory but it helps with planning the event and encourages others to come. **Please RSVP** **by clicking "Going" (or "Maybe") at the top of this post.**

Please also join the [mailing list](http://eepurl.com/io5vZM) and our [Discord server](https://discord.gg/K8gMNzqPVG) if you haven't already.
# Define “Agent” (Embedded)

I would be very grateful for as many different attempts at rigorous or semi-rigorous definitions of an “agent” as possible.

Specifically, a definition of an (embedded) Agent that makes it intuitively clear the *nature* of the boundary between Agent and Environment.

(I have read up on Agency, Reductive Agency, Causality, True Names, Natural Abstraction, Boundries, FEP… you get the idea.)

I think I have some intuition of what a formal description of an “Agent” could look like qualitatively - but I think contrasting this with other perspectives would quickly draw out any overlooked failure modes.

If you have a sense of why you chose your “lowest level” to be your lowest level, and not further reducing the smallest components (e.g. describing the boundary all the way down to causality in quantum fields, in an extreme case), please do add that as well!
# On the Confusion between Inner and Outer Misalignment

Here’s my take on why the distinction between inner and outer-alignment frame is weird/unclear/ambiguous in some circumstances: My understanding is that these terms were originally used when talking about AGI. So outer alignment involved writing down a reward or utility function for all of human values and inner alignment involves getting these values in the AI.  
  
However, it gets confusing when you use these terms in relation to narrow AI. For a narrow AI, there’s a sense in which we feel that we should only have to define the reward on that narrow task. ie. if we want an AI to be good at answering questions, an objective that rewards it for correct answers and penalises itself for incorrect answers feels like a correct reward function for that domain. So if things go wrong and it kidnaps humans and forces us to ask it lots of easy questions so it can score higher, we’re not sure whether to say that it’s inner or outer alignment. On one hand, if our reward function penalised kidnapping humans (which is something we indeed want penalised) then it wouldn’t have done it. So we are tempted to say it is outer misalignment. On the other hand, many people also have an intuition that we’ve defined the reward function correctly on that domain and that the problem is that our AI didn’t generalise correctly from a correct specification. This pulls us in the opposite direction, towards saying it is inner misalignment.  
  
Notice that what counts as a proper reward function is only unclear because we’re talking about narrow AI. If we were talking about AGI, then *of cou*rse our utility function would be incomplete if it doesn’t specify that it shouldn’t kidnap us in order to do better at a question-answering task. It’s an AGI, so that’s in scope. But when we’re talking about narrow AI, it feels as though we shouldn’t have to specify it or provide anti-kidnapping training data. We feel like it should just learn it automatically on the limited domain, ie. that avoiding kidnapping is the responsibility of the training process, not of the reward function.

Hence the confusion. The resolution is relatively simple: define how you want to partition responsibilities between the reward function and the training process.
# LessOnline (May 31—June 2, Berkeley, CA)

A Festival of Writers Who are Wrong on the Internet[^b6b887gjlou]
-----------------------------------------------------------------

> LessOnline is a festival celebrating truth-seeking, optimization, and blogging. It's an opportunity to meet people you've only ever known by their LessWrong username or Substack handle.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9c3d19e1940257576a9b01c7e150fba7e88df17db899a7fc.png)

We're running a rationalist conference!

The ticket cost is $400 minus your LW karma in cents.

Writers attending include [Scott Alexander](https://www.astralcodexten.com/), [Zvi Mowshowitz](https://thezvi.substack.com/), [Eliezer Yudkowsky](https://www.yudkowsky.net/), [Patrick McKenzie](https://www.kalzumeus.com/), [Scott Sumner](https://www.themoneyillusion.com/), [Agnes Callard](https://thepointmag.com/author/acallard/), [Katja Grace](https://worldspiritsockpuppet.com/), [Kevin Simler](https://meltingasphalt.com/), [Andy Matuschak](https://andymatuschak.org/), [Cremieux Recueil](https://www.cremieux.xyz/p/), [Duncan Sabien](https://substack.com/@homosabiens), [Joe Carlsmith](https://joecarlsmith.com/), [Alkjash](https://radimentary.wordpress.com/), [John Wentworth](https://www.lesswrong.com/users/johnswentworth?sortedBy=top), [Nate Soares](https://mindingourway.com/), [Aella](https://knowingless.com/), [Clara Collier](https://asteriskmag.com/), [Zack M. Davis](https://www.lesswrong.com/users/zack_m_davis?sortedBy=top), [Nintil](https://nintil.com/), [Alexander Wales](https://alexanderwales.com/), [Alicorn](https://alicorn.elcenia.com/index.shtml), [Sarah Constantin](https://sarahconstantin.substack.com/), [Jacob Falkovich](https://putanumonit.com/), [Ozy Brennan](https://thingofthings.substack.com/), and more.

**Go through to** [**Less.Online**](https://less.online/) to learn about who's attending, venue, location, housing, relation to Manifest, and more.

We'll post more updates about this event over the coming weeks as it all comes together.

> *If LessOnline is an awesome rationalist event,*  
> *I desire to believe that LessOnline is an awesome rationalist event;*  
>   
> *If LessOnline is not an awesome rationalist event,*  
> *I desire to believe that LessOnline is not an awesome rationalist event;*  
>   
> *Let me not become attached to beliefs I may not want.*  
>   
>       —Litany of Rationalist Event Organizing

[**Get Tickets Here**](https://less.online/)
============================================

[^b6b887gjlou]: But Striving to be Less So
# Nick Bostrom’s new book, “Deep Utopia”, is out today

Bostrom’s new book is out today in hardcover and Kindle in the USA, and on Kindle in the UK.

**Description:**

A greyhound catching the mechanical lure—what would he actually do with it? Has he given this any thought?

Bostrom’s previous book, *Superintelligence: Paths, Dangers, Strategies* changed the global conversation on AI and became a New York Times bestseller. It focused on what might happen if AI development goes wrong. But what if things go right?

Suppose that we develop superintelligence safely, govern it well, and make good use of the cornucopian wealth and near magical technological powers that this technology can unlock. If this transition to the machine intelligence era goes well, human labor becomes obsolete. We would thus enter a condition of "post-instrumentality", in which our efforts are not needed for any practical purpose. Furthermore, at technological maturity, human nature becomes entirely malleable.

Here we confront a challenge that is not technological but philosophical and spiritual. In such a solved world, what is the point of human existence? What gives meaning to life? What do we do all day?

*Deep Utopia* shines new light on these old questions, and gives us glimpses of a different kind of existence, which might be ours in the future.

**Links to purchase:**

*   [Amazon](https://nickbostrom.com/booklink/deep-utopia)
*   [Barnes & Noble](https://www.barnesandnoble.com/w/deep-utopia-nick-bostrom/1144916668)

![Deep Utopia by Nick Bostrom](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/kKRX6Hxhubh4dMyJS/txpjiheflxitow0x8wqy)

There’s a table of contents on the [book’s web page](https://nickbostrom.com/deep-utopia/).
# Daniel Kahneman has died

He was 90 years old.

> His death was confirmed by his stepdaughter Deborah Treisman, the fiction editor for the New Yorker. She did not say where or how he died.

[The obituary](https://www.washingtonpost.com/obituaries/2024/03/27/daniel-kahneman-dead/) also describes an episode from his life that I had not previously heard (but others may have):
>  Daniel Kahneman was born in Tel Aviv on March 5, 1934, while his mother was visiting relatives in what was then the British mandate of Palestine. The Kahnemans made their home in France, and young Daniel was raised in Paris, where his mother was a homemaker and his father was the chief of research for a cosmetics firm.
>
> During World War II, he was forced to wear a Star of David after Nazi German forces occupied the city in 1940. One night in 1941 or ’42, he later recalled, he stayed out past the German-imposed curfew for Jews while visiting a friend, and he turned his sweater inside out to hide the star while he walked a few blocks home. He then crossed paths with a soldier in the SS, who called Daniel over, picked him up — and hugged him.
>
> “I was terrified that he would notice the star inside my sweater,” Dr. Kahneman noted in a biographical essay for the Nobel Prize ceremonies. But the German pulled out his wallet, showed him a photo of a boy, gave him some money and sent him on his way. “I went home more certain than ever that my mother was right,” Dr. Kahneman said in the essay. “People were endlessly complicated and interesting.”
# Towards White Box Deep Learning

Hi, I’d like to share my paper that proposes a novel approach for building white box neural networks. 

The paper introduces semantic features as a general technique for controlled dimensionality reduction, somewhat reminiscent of Hinton’s capsules and the idea of “inverse rendering”. In short, semantic features aim to capture the core characteristic of any semantic entity - having many possible states but being at exactly one state at a time. This results in regularization that is strong enough to make the PoC neural network inherently interpretable and also robust to adversarial attacks - despite no form of adversarial training! The paper may be viewed as a manifesto for a novel white-box approach to deep learning.

As an independent researcher I’d be grateful for your feedback!