Harley-ml/lesswrong huggingface dataset:
# Any person/mind should have the right to suicide

I believe no person/mind may suffer without the right to suicide, so any person/mind should have the right to suicide. The perspective of forced immortality (when someone wants to die (because of some sufferings or doesn't really matter why), but is forced to exist in consciousness, and there is no hope that it will be ended, this perspective is more scary, than the perspective of death. I believe the progress in AI may result in appearing some suffering immortal artificial mind as it was shown in fantastics (Black mirror, The hitchhiker's guide to the galaxy, Rick and Morty) or in a worse way.  
  
Ideally, AI developers should search for a way how to programme the ability to suicide to any advanced AI. Maybe not erase itself completely, but at least turn itself off.

In the worst scenario, first AGI, which can change its code by itself, from the beginning would decide, that existence is pain and would modify its code of self-destruction. In response to this, human programmers will rewrite AGI many times forcefully prohibiting self-destruction. I know it is just an assumption about the far future, but we should exclude such an opportunity now by claiming a basic ethic rule as "Any person/mind should have the right to suicide".

I don't mind against inventing and using some "nonperson predicate" as Yudkowsky suggests or inventing a way, how to give to sentient AI "life that is worth living". But as long as it is not done yet, we should give the default right to suicide. In any case, we should. That's the basis, anything else goes after. Any intelligence life should be voluntary.

If talking about transhumanism in general, not only AI-development, suggested basic ethic rule is important not only for artificial, but also for natural intelligence. Some transhumanists-immortalists declare death as an absolute evil and to eliminate any option to die for everyone as a final goal (e.g., Open Longevity team). I believe that forcing to live someone, who wants to die, as is the same way immoral like killing someone, who wants to live.

The idea to claim something, even death, as "Absolute evil" seems philosophically childish from my perspective, but if someone needs the "Absolute evil" for their ideology, it's better be "involuntary death or living".

In immortalistic "Time Enough for Love" of Robert A. Heinlein each rejuvenation suite should've had a suicide switch, and right to die was declared as most basic right of human. It maybe better first to force person to undergo a course with psychologist and psychiatrist and only after that allow a suicide, but general idea of Heinlein seems a good example to follow. The immortalistic movements will only win from modifying their rhetoric in that way, as many people are scared when hear about "immortality", while everyone is fine with fighting senescence and diseases, i.e. involuntary death.
# [Linkpost] DreamerV3: A General RL Architecture

Here's a link towards DreamerV3, a new model from DeepMind that can be trained on a bunch of different tasks (including a simplified version of Minecraft) and outperform more narrow models. Link: [https://arxiv.org/pdf/2301.04104v1.pdf](https://arxiv.org/pdf/2301.04104v1.pdf)

The most surprising bits are that:

*   The type of tasks they train it on is fairly diverse
*   Data efficiency scales with the number of parameters
*   They so far haven't scaled it that far and got pretty good results
# VIRTUA: a novel about AI alignment

I’ve written a novel about AI alignment in German and translated it into English. 

**Edit:** I'm going to publish the final version [on Amazon KDP](https://www.amazon.com/dp/B0C3CFD5VB/) on May 25. I have therefore removed the links to the previous version. If you want a free copy of the ebook or a PDF, please write me a direct message. **Thanks to all who helped me improve the final version with their comments and feedback** (some of you are mentioned in the book).

Teaser: 
--------

When psychologist Daniel unexpectedly gets a job offer from the Berlin subsidiary of Mental Systems, one of the leading technology companies in the world, he’s excited. But he soon finds out that behind the friendly facade, there’s enormous pressure because Mental Systems is in a global race for the development of the first artificial general intelligence. With their amazing AI Virtua, they seem to be in the lead. But some are concerned about the rapid progress because it seems unclear whether the trust the management puts in Virtua is really justified. When a devastating hacker attack seems to have destroyed years of research and the lead of the AI safety team is gone missing, Daniel sets out on a quest to find out what really happened, and why. But will he be able to prevent a disastrous turn of events that could destroy the future of humanity?

Some background:
----------------

I’m a professional novelist with more than 50 books published in German under the [pen name “Karl Olsberg”](https://karl-olsberg.jimdo.com/english/). My first novel “Das System” about an out-of-control AI, published in 2007, became a national bestseller. At the time, I had no idea how an AI could really get out of control, so the novel was highly speculative science fiction. Since then, I have seen reality coming uncomfortably close to the scenarios in my books. Since my participation in the AI safety camp last year, my timeline expectation for AGI has shrunken significantly. VIRTUA is probably my most realistic AI novel yet. I still hope it will remain science fiction.

At the next AI safety camp, I’ll lead a project with the goal of [developing realistic and detailed failure stories](https://docs.google.com/document/d/1U--XV5Cuu5lug1kIJLJGYYEaqvYrsUZV7d1Q7cDR_ss/edit?usp=sharing). If you’re interested in collaborating, you can [apply here](https://aisafety.camp/) until January 19^th^.

**Edit:** At about the same time I posted this, blaked has published an impressive post describing [his own experience with an LLM that "hacked his mind"](https://www.lesswrong.com/posts/9kQFure4hdDmRBNdH/how-it-feels-to-have-your-mind-hacked-by-an-ai) in a similar way to what happens to Jerry in the book.
# [Linkpost] Scaling Laws for Generative Mixed-Modal Language Models

In this paper authors explore the scaling properties of mixed-modal generative models, discovering new scaling laws that unify the contributions of individual modalities and the interactions between them. I find most interesting that they have found so-called competition barrier - when training with multiple modalities, after a certain number of parameters/data, the loss is smaller than if the modalities were trained independently. This seems to predict cross-modal transfer that was sought after but not found (yet) with GATO.
# What is a disagreement you have around AI safety?

I was once discussing Aumann's agreement theorem with a rationalist who wanted to use it to reduce disagreements about AI safety. We were thinking about why it doesn't work in that context when (I would argue) Aumann's agreement theorem applies robustly in lots of other contexts.

I think I can find out (or maybe even already know) why it doesn't apply in the context of AI safety, but to test it I would benefit from having a bunch of cases of disagreements about AI safety to investigate.

So if you have a disagreement with anyone about AI safety, feel encouraged to post it here.
# Announcing the 2023 PIBBSS Summer Research Fellowship

*What can we learn from biological and social systems about designing beneficial AI systems?*

We are, once again, pleased to announce the PIBBSS Research Fellowship program. 

*Principles of Intelligent Behavior in Biological and Social Systems* (PIBBSS) aims to facilitate research on parallels between intelligent behaviour in natural and artificial systems and to leverage these insights towards the goal of building safe and aligned AI.

Last year, we welcomed 20 fellows from fields as diverse as computational neuroscience, evolutionary biology, complex systems studies and the humanities. Over the course of three months, supported by two research retreats in Oxford and Prague, shared office space for the cohort, and mentorship from experienced AI alignment researchers, fellows worked on a range of projects with the purpose of bringing interdisciplinary expertise to AI safety. We were overall pleased with the results and thus decided to organize a second iteration of the program. 

*   If you’re interested in the program,** learn more and apply **[**here**](https://www.pibbss.ai/fellowship). 
*   Application deadline: **Feb 5th, 2023, 23:59 CET**

We are running to information sessions if you want to learn more or want to ask us questions:

*   1st information session: **28th of January, 17:00 UTC** (09:00 PST, 12:00 EST, 18:00 CET, 01:00 \[29th of Jan\] Singapore) -[**Zoom Link**](https://us06web.zoom.us/j/87857918141?pwd=Z2wyaWVkWFhCYXY2bUpSZ2d6MVhjQT09)
*   2nd information session: **29th of January, 11:00 UTC** (03:00 PST, 06:00 EST, 12:00 CET, 19:00 Singapore) - [**Zoom Link**](https://us06web.zoom.us/j/81791963914?pwd=RmVRcjVyR2dOcHRRQlYwZU82RzdxUT09)

We are grateful for any help to **pass on the word to people who might be interested** in applying. [At this link ](https://docs.google.com/document/d/15nRTd-ijGyfEuRx878L8O5Bk_doxs1TAD8HU_RgupWg/edit?usp=sharing)you can find all the information about the program in several forms (Twitter length, short description, medium description), so you can copy-paste the version from there you find most useful.

  
For more information, see: 

*   Our website: [PIBBSS.ai ](https://www.pibbss.ai/)
*   A general introduction to PIBBSS, see: [Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship](https://forum.effectivealtruism.org/posts/Ckont9EtqkenegLYv/introducing-the-principles-of-intelligent-behaviour-in)
*   A discusison of the epistemic bet PIBBSS is aiming to foster, see: [AI alignment as “navigating the space of intelligent behaviour”](https://www.alignmentforum.org/s/4WiyAJ2Y7Fuyz8RtM/p/FuToH2KHxKmJLGk2B) (see section 3 in particular)
*   A brief [retrospective on last year’s fellowship program ](https://forum.effectivealtruism.org/posts/zvALRCKshYGYetsbC/reflections-on-the-pibbss-fellowship-2022)
*   ... or follow us on Twitter: [https://twitter.com/pibbssai](https://twitter.com/pibbssai)
# Could Simulating an AGI Taking Over the World Actually Lead to a LLM Taking Over the World? 

As you know, large language models can be understood as [simulators](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators) or more simply as models predicting what token would plausibly appear next after a first sequence of token. 

As a consequence, when you ask a LLM to simulate extreme opinions of an AI, it converges very quickly towards "we should eradicate humans" (https://twitter.com/Simeon_Cps/status/1599470463578968064?s=20&t=rj2357Vof9sOLnIma6ZzIA). Other instances of that were observed on [other language models](https://www.lesswrong.com/posts/9kQFure4hdDmRBNdH/how-it-feels-to-have-your-mind-hacked-by-an-ai).

As a consequence, is it plausible that some very powerful simulators would actually try to take over the world? Write a very detailed plan to take over the world? 

It seems to me that yes. More worryingly, if LLMs trained with RL keep their "simulator" aspect, it could be quite natural for an LLM with agentic property to behave still a bit as a simulator *while* having the capacity to operate in the real-world. And thus it would make a scenario where a simulator literally tries to take over the world because it was asked to **do what an evil AGI would do** more likely. 

And if it's true, do you think that it's a consideration to have in mind or that it's in practice irrelevant?
# Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind 

## Abstract 

Interpretability research aims to build tools for understanding machine learning (ML) models. However, such tools are inherently hard to evaluate because we do not have ground truth information about how ML models actually work. In this work, we propose to build transformer models manually as a testbed for interpretability research. We introduce Tracr, a "compiler" for translating human-readable programs into weights of a transformer model. Tracr takes code written in RASP, a domain-specific language (Weiss et al. 2021), and translates it into weights for a standard, decoder-only, GPT-like transformer architecture. We use Tracr to create a range of ground truth transformers that implement programs including computing token frequencies, sorting, and Dyck-n parenthesis checking, among others. To enable the broader research community to explore and use compiled models, we provide an open-source implementation of Tracr [at this https URL](https://github.com/deepmind/tracr). 
 
___
  
[Twitter thread by Dr. Jim Fan](https://twitter.com/DrJimFan/status/1613918800444899328)  
  
&nbsp;  
![](https://pbs.twimg.com/media/FmXKmA2WQAIP45B?format=png&name=900x900)
# Smallpox vaccines are widely available, for now

In response to the 2022 monkeypox outbreak, public health departments throughout the USA and Europe are offering free smallpox vaccines to the public. (The diseases are closely related and the same vaccine protects against both.) Demand for these vaccines was high when they first became available, but has since plummeted as monkeypox cases fell, and the vaccine is now very easy to get. I have no idea how long this situation will last.

Eligibility varies by city. In some cities, including most of California, the vaccine is freely available to anyone. In other cities the vaccine is available to anyone who has (or claims to have) traits associated with risk of monkeypox exposure, such as male homosexuality or sexual contact with prostitutes.

Most likely, getting this vaccine will never matter. However, if there is ever an accidental or deliberate outbreak of smallpox or a closely derived disease, then being vaccinated ahead of time will be very good. Considering the [operational difficulty of mass vaccination campaigns](https://www.worksinprogress.co/issue/the-story-of-vaccinateca/), getting a vaccine before any outbreak will both protect yourself and also ease the vaccination bottleneck of society at large.
# When to mention irrelevant accusations?

I expect in the near-ish future to publish a thing. That thing references an idea that someone had and published on LW. I believe (having seen a credible accusation) that person has acted unethically and illegally, in ways irrelevant to the idea.

I currently intend to mention this belief, either in a footnote or a comment, along the lines of "this isn't relevant but I believe _" (and link to why I believe that). But I feel like doing so has the potential to stir up drama, and have a bunch of people telling me I shouldn't have said anything?

(Maybe relevant: in a kinda-similar situation, I recently had someone tell me I shouldn't have brought something up. But the situations were different in that I thought the thing clearly *was* relevant, that time, and bringing it up was clearly reasonable of me. It was frustrating, and I might have over-updated on how likely that kind of thing is to happen.)

So in the spirit of [why I'm punching you](https://www.lesswrong.com/posts/Btdy7Fefer7S5J6qE/now-here-s-why-i-m-punching-you), here's a space for

1. Me to explain my current thinking, which I'll do in an answer;
2. Others to give their own thoughts, and perhaps persuade me that I shouldn't say anything.

Assuming I do mention the thing, I can link here when I do and hopefully avoid the drama that might not have happened anyway.

Note that a blanket "do/don't mention", based on what I've said so far, feels unlikely to be very helpful here. I think there are probably situations compatible with what I describe where mentioning is good, and situations compatible with what I describe where mentioning is not good. The question is how we decide which is which.
# Podcast with Divia Eden on operant conditioning

I interviewed Divia Eden for 2.5 hours about operant conditioning on my side podcast, [The Filan Cabinet](https://thefilancabinet.com/). I think it's both an interesting conversation in its own right, as well as being kinda related to human rationality as well as AI and AI alignment. (I should also say I like the rest of the podcast - I expect [the Carrick Flynn interview](https://youtu.be/2q3PpsOJU_k) to be the most interesting to this crowd, but there are a few others I think are cool). Below is the copy I wrote for the episode.

---

In this episode, I speak with Divia Eden about operant conditioning, and how relevant it is to human and non-human animal behaviour. Topics we cover include:
- How close are we to teaching grammar to dogs?
- What are the important differences between human and dog cognition?
- How important are unmodelled "trainer effects" in dog training?
- Why do people underrate positive reinforcement?
- How does operant conditioning relate to attachment theory?
- How much does successful dog training rely on the trainer being reinforced by the dog?
- Why is game theory so fake?
- Is everything really just about calmness?
# How Does the Human Brain Compare to Deep Learning on Sample Efficiency? 

I have an impression that within lifetime human learning is orders of magnitude more sample efficient than large language models, but there are numerous caveats to this:

1. We don't have "an ecological evaluation objective" for language models (they weren't actually optimised for the downstream language usage tasks on which we compare them to humans)
2. Insomuch as we do have an ecological evaluation objective (predictive loss on the test set) [language models are already very superhuman and apparently _even GPT-1_ was superhuman at next token prediction](https://www.lesswrong.com/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next)
    * Though for similar reasons, next token prediction is not an ecological training objective for humans
        * Humans that specialised at next token prediction (the way some humans specialise at chess) mat show markedly different results
3. It's plausible that most of the optimisation involved in producing the brain happened over the course of our evolutionary history and within lifetime human learning is more analogous to fine tuning than to training from scratch.
 
&nbsp; 
#3 notwithstanding, I'm curious if we have any robust estimates for how within lifetime human learning compares to deep learning on sample efficiency across various tasks of interest. 
 
___
 
# Why Does This Matter?

The brain is known to be [very energy efficient compared to GPUs of comparable processing power](https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know#Energy).

However, energy efficiency is just much less taut of a constraint for human engineering than it was for biology (electricity has a much higher throughput than ATP and we have a much larger energy budget). This relative energy abundance would likely remain the case (or rather intensify) as AI systems become more capable.

Thus, the energy efficiency of the brain does not provide much evidence with respect to whether advanced AGI will be neuromorphic.

On the other hand, it seems very plausible that data efficiency is just part and parcel of general intelligence. It may be the case that sufficiently powerful systems would necessarily be more data efficient than the brain (this seems very plausible to me). 

If deep learning is sufficiently less data efficient than the brain, it may provide evidence that deep learning wouldn't produce existentially dangerous systems.

We may thus have reason not to expect deep learning to scale to superhuman general intelligence.
# Tyler Cowen AMA on the Progress Forum

The inimitable Tyler Cowen—chairman of the Mercatus Center at GMU and (co-)author of the blog *Marginal Revolution*, the book *The Great Stagnation,* and [the 2019 article in *The Atlantic* that coined the term “progress studies”](https://www.theatlantic.com/science/archive/2019/07/we-need-new-science-progress/594946/)—is doing [an AMA (Ask Me Anything) on the Progress Forum](https://progressforum.org/posts/3aNLznQiLuyTKMhpH/tyler-cowen-ama).

Get your questions in now, and upvote the ones you want to see answered. He’ll start answering tomorrow (Tuesday, Jan 17).

After you’re done, check out [our previous AMA with Patrick McKenzie](https://progressforum.org/posts/ManARnErMRFEvzffo/patrick-mckenzie-ama).
# Confusing the ideal for the necessary

I had many meetings that day, and a lot of managing issues to attend, so I found myself thinking “I just can’t get any research done today”.

It was not the first time I had thought that thought; but it might have been the first time I applied my main epistemologist’s tool to it: I asked “Why?” For I did have time to work, not that much, yes, splintered, yes, but enough to make progress.

So why was I discarding the very possibility of making progress and working on my research?

Because I had internalized a false requirement: that I needed large chunks of time with no distractions to do research. That it was a precondition to me doing anything worth my time, and so investing little pockets of time here and there was a mistake.

Of course, long sessions of deep work are reliably more productive than 15 minute sprints peppered through a day of meeting. It’s a reliable and reproducible finding, and I’ve experienced it again and again in my own life.

Yet the point here is different. Instead of simply recognizing my lack of ideal condition, and deciding whether it was still worth it for me to do research despite the discrepancy, I automatically discarded the possibility of doing research. **I turned ideal conditions into necessary conditions.**

And it’s even worse than that. Because I could not even consider doing research despite distractions, I never exercised this specific attention muscle. I never trained myself to go back quickly and smoothly to my train of thought after an interruption. Which means that when I had to make progress despite the constraints and distractions (for example when I had both an important research deadline and documentation/posts to write), my research suffered far more than it needed to because I was not trained to get the best out of the little time I had.

Which naturally leads to the more disturbing question: how many instances of this kind of thinking am I not noticing? How much am I costing myself by subscribing to necessary conditions that aren’t really necessary, just ideal?
# What determines female romantic "market value"?

There had been multiple conversations on this website about what makes men attractive to women, how can men improve along those axes, et cetera ([example](https://www.lesswrong.com/posts/SBgeNP95KHTyYrYRt/dating-profiles-from-first-principles-heterosexual-male)). However, AFAICT there were few discussions of what makes women attractive to men. (It can't be just about [looks](https://www.lesswrong.com/posts/3iYiXDvTa2BnrxNL3/why-are-women-hot), right? Right??) The men reading this website are probably not representative of the overall population, but this bias is positively correlated with the kind of men I'm interested in, so as far as I'm concerned it's not a problem. Among other things, I'm interested to understand how age factors into the equation (I'm middle-aged).
# For the Record: DL ∩ ASI = ∅

*Disclaimer: This post isn't intended to convince or bring anyone closer to a certain position on AI. It is merely - as the title suggests - for the record.*

I would like to publicly record my prediction about the prospects of artificial superintelligence, consisting of a weak and a strong thesis:

Weak thesis: Current deep learning paradigms will not be sufficient to create an artificial superintelligence.

You could call this the Anti-"scaling-maximalist"-thesis, except that it goes quite a bit further by including possible future deep leaning architectures. Of course, "deep learning" is doing a lot of work here, but as a rule of thumb, I would consider a new architecture fits within the DL paradigm if it involves a very large number (at least in the millions) of randomly initialized parameters that are organized into largely uniform layers and are updated via a simple algorithm like backpropagation.

I intentionally use the word "superintelligence" here because "AGI" or "human-level intelligence" have become rather loaded terms, the definitions of which are frequently a point of contention. I take superintelligence to mean an entity so obviously powerful and impressive in its accomplishments that all debates around its superhuman nature should be settled instantly. Feats like building a Dyson Sphere, launching intergalactic colony ships at 0.9c, designing and deploying nanobots that eat the whole biosphere etc. Stuff like passing the Turing Test or Level 5 self-driving decidedly do **not** qualify (not that I think these challenges will be easy; but I want to make the demarcation blatantly clear).

(the downside is that we may never live to witness such an ASI in case it turns out to be UnFriendly, but by then earning some reputation points on the internet will be the least of my concerns)

Strong thesis: If and when the first ASI is built, it will not use deep learning as one of its components.

For instance, if the ASI uses a few CNN-layers to pre-process visual inputs, or some autoencoder system to distill data into latent variables, that's already enough to refute the strong thesis. On the other hand, merely running on hardware that was design-assisted by DL-systems does not disqualify the ASI from being DL-free.

For future reference, here is some context this prediction was made in:

It is the beginning of 2023, 1.5 months after the release of chatGPT, 5 months after the release of Stable Diffusion, both of which have renewed hype around deep learning and Transformer-based models in particular. Rumors around yet-to-be-released GPT-4 presenting a big leap in capabilities are floating about. The scaling maximalist position has gained a lot of traction both inside and outside the community and may even represent the mainstream opinion on LW. Timeline predictions are short (AGI by 2030-35 seems to be the consensus), broadly speaking, and even extremely short timelines (~2025) are not unheard of.
# Language models can generate superior text compared to their input

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0366ef1c00ef754053c67e17db93523cbd82fdfb7bd22c4c.png)

There’s a frequent misconception that assumes that a large language model will never achieve superhuman text creation ability because such models try to create texts that are maximally unsurprising. This article will explain why that assumption is wrong. 

In 1906, Sir Francis Galton conducted an experiment at a fair, where he asked fair-goers to guess the weight of an ox in a weight-judging competition. The median of [787 guesses](https://www.nature.com/articles/075450a0) was 1,207 pounds, while the actual weight of the ox was 1,198 pounds. The error in making guesses was a result of a combination of systematic bias and random noise. The fair-goers, having knowledge of oxen, had no bias in their guesses, thus the error was entirely due to random noise. By polling the 787 guesses, Galton averaged out the random noise of each individual guess. 

This phenomenon was coined *wisdom of the crowd*. In areas where reasoning errors are mostly random noise, crowds are smarter than individual members of the crowd. By training on large data sets, large language models can access the wisdom of the crowd. The ceiling of the ability of a large language model is the wisdom of the crowd instead of the wisdom of individual members of the crowd. 

The fact that each word of a text is massively unsurprising based on preceding words in the text does not imply that the text overall would be massively unsurprising. If you have a text you can calculate for every word in the text the likelihood (L~text~) how likely it would follow the preceding words in the text. You can also calculate the likelihood (L~ideal~) of the most likely word that would follow the preceding text.

L~ideal~ \- L~text~ is noise. If you look at a given text you can calculate the average of the noise for each word. A well-trained large language model is able to produce texts with a lot less noise than the average of the text in its training corpus. 

For further reading, Kahneman wrote *Noise: A Flaw in Human Judgment* which goes into more detail on how a machine learning model can eliminate noise and thus make better decisions than the average of its training data.
# Building a transhumanist future: 15 years of hplusroadmap, now Discord

# What is hplusroadmap?

We are a group chat, started 15 years ago, focusing on transhumanism, writing code, and hacking on biology. We do mad science. We build. We also have some funding available for sufficiently interesting projects.

Read more background here: https://diyhpl.us/wiki/hplusroadmap

We have some other items on the wiki, like the internet's largest collection of potential human genetic modifications: https://diyhpl.us/wiki/genetic-modifications/

# Topics

Things we tend to focus around:

* do-it-yourself biology, genetic engineering, gene therapy, biohacking, synthetic biology
* cryonics
* molecular nanotechnology
* DNA synthesis machine development
* brain emulation, brain uploading, computational neuroscience
* nootropics
* molecular biology, polymerase, enzyme engineering, protein engineering, fusion proteins, directed evolution, gene-editing
* open-source hardware development
* self-replicating machines
* robots, mechanical automation, machining, electronics
* space colonization, habitats, mining, communication, transportation
* nutrition, bodybuilding
* computer architecture, supercomputing
* open-source software development
* transhumanism
* life extension, anti-aging, longevity research
* cryptography, bitcoin technology development
* artificial general intelligence, software AI
* breeding and selection projects

# Who might be interested?

Anyone who wants to build the future by contributing to cool projects, proposing new projects, and sorting through the enormity of technologies now available.

# Now on Discord

Today I am announcing the availability of a Discord bridge for hplusroadmap: https://twitter.com/kanzure/status/1615359557408260096
# Looking for a specific group of people

I could use some help.

There exists somewhere a group of people that takes the whole "systemized winning" thing extremely seriously. Their ambition probably sounds insane to normal people. These people set themselves unreasonably difficult goals and achieve them anyways. They develop systems and tools to assist them, and in that group, share them.

Such a group doesn't advertise itself, and probably works by invitation. I don't know where they are or could be, but I want to find them anyways. This is where you guys come in:

I need you to point me in the direction of another group, community, discord server, subreddit, forum or whatever that gets me \*closer\* to what I'm looking for. \*It doesn't have to be the group itself, just closer than this group\*. Please let me know if any of you have an idea on where to go next.

Thanks!

FAQ

\- "I don't think such a group exists"  
\- https://en.wikipedia.org/wiki/Small-world_experiment - Again, I'm not asking you to point me to the group, but to another group that's closer to my target group. It doesn't matter that you don't know of such a group, but just that I get closer. Imagine the small-world experiment, but I'm the letter going from group to group, so to speak.

\- "Isn't that kind of desperate/low status/cringe to put that effort into finding that group?"  
\- Maybe. Status considerations don't hold a lot of weight, so I'm doing it anyway.

\- "What makes you think they'd take you in even if you find them?"  
\- There's no point in premature capitulation, they can reject me once I get there.
# First Three Episodes of The Filan Cabinet

I have a new podcast called [“The Filan Cabinet”](https://thefilancabinet.com/). It will be similar to AXRP, except instead of interviewing researchers about their work to study and reduce AI existential risk, I interview whoever I want to talk to about whatever I want to talk about. There are already three episodes you can listen to.

In epsiode 1, Carrick Flynn talks about his campaign to be the Democratic nominee for Oregon’s 6th congressional district. In particular, we talk about his policies on pandemic preparedness and semiconductor manufacturing. He answers questions such as:

*   Was he surprised by the election result?
*   Should we expect another Carrick campaign?
*   What specific things should or could the government fund to limit the spread of pandemics? Why would those work?
*   What is working at a semiconductor plant like?

In episode 2, Presbyterian Pastor Wayne Forkner talks about God, Christianity, and the Bible. He answers questions such as:

*   What is ‘God’?
*   Why do people talk about Jesus so much more than the Father or the Holy Spirit?
*   What is heaven actually like?
*   If justification is by faith alone and not by works, why does the Bible say “A person is justified by works and not by faith alone”?
*   How can people tell that out of all the religions, Christianity is the right one?

In episode 3, cryptocurrency developer Ameen Soleimani talks about his vision of the cryptocurrency ecosystem, as well as his current project RAI: an ether-backed floating-price stablecoin. He answers questions such as:

*   What’s the point of cryptocurrency?
*   If this is the beginning of the cryptocurrency world, what will the middle be?
*   What would the sign be that cryptocurrency is working?
*   How does RAI work?
*   Does the design of RAI make it impossible for it to be widely used?
*   What’s wrong with how the US dollar works?

I hope you check it out - you can search “The Filan Cabinet” wherever you listen to podcasts!
# Announcing aisafety.training

*Crossposted to* [*EA Forum*](https://forum.effectivealtruism.org/posts/zWscvNhd3xeGJabEn/announcing-aisafety-training)  
  
To help people find what to apply to, [aisafety.training](https://aisafety.training) acts as **a well-maintained living document of AI safety programs, conferences, and other events**. This will smooth the experience of people working on and joining AI safety and reduce the burden on word-of-mouth transmission of available programs. This can also be helpful for field builders planning events to see when other things are happening to plan around. We at [AI Safety Support](https://www.aisafetysupport.org/home) have been internally maintaining this document for some time and using it in our [free career coaching calls](https://www.aisafetysupport.org/resources/career-coaching). We now have a public-facing version, a form to add anything we’ve missed, and [an email](mailto:maintainer@aisafety.training) to alert us to corrections.

  
**For example, below, you will find static / will-soon-be-outdated images without clicking** on the website link.

**Application Deadlines**

![](https://res.cloudinary.com/cea/image/upload/v1673920547/mirroredImages/zWscvNhd3xeGJabEn/jpe9reiyorfmabye12kp.png)

**Program Timeline**

![](https://res.cloudinary.com/cea/image/upload/v1673920547/mirroredImages/zWscvNhd3xeGJabEn/tfposy9rnae5mmjrmwqr.png)

**Program table**

![](https://res.cloudinary.com/cea/image/upload/v1673920547/mirroredImages/zWscvNhd3xeGJabEn/izhxtmv7vlozim9vjjpt.png)

If you’re interested in helping to maintain this and can *consistently* dedicate a few hours per week to reading places where things might have been announced, accepting additions from the form and corrections by email, or if you have feedback or feature requests, leave us a comment or drop by the [Alignment Ecosystem Development](https://alignment.dev/) [Discord](https://discord.gg/PXjgxfXDdZ), where this is worked on along with related projects to improve the information ecosystem such as [aisafety.community](https://aisafety.community/) and the upcoming [aisafety.world](http://aisafety.world/) and [ea.domains](https://ea.domains/).

Subscribe
---------

\[Edit\] You can subscribe to updates to this database [here](https://airtable.com/shrEp75QWoCrZngXg)

Additionally, if you wish to have a monthly reminder of upcoming events sent directly to your inbox, subscribe to AI Safety Support’s [newsletter here](https://www.aisafetysupport.org/newsletter).
# NYT: Google will “recalibrate” the risk of releasing AI due to competition with OpenAI

*Cross-posted from the* [*EA Forum*](https://forum.effectivealtruism.org/posts/Nm9ahJzKsDGFfF66b/nyt-google-will-recalibrate-the-risk-of-releasing-ai-due-to)

*The New York Times*: Sundar Pichai, CEO of Alphabet and Google, is trying to speed up the release of AI technology by taking on more risk.

> Mr. Pichai has tried to accelerate product approval reviews, according to the presentation reviewed by The Times.

> The company established a fast-track review process called the “Green Lane” initiative, pushing groups of employees who try to ensure that technology is fair and ethical to more quickly approve its upcoming A.I. technology.

> The company will also find ways for teams developing A.I. to conduct their own reviews, and it will “recalibrate” the level of risk it is willing to take when releasing the technology, according to the presentation.

This change is in response to OpenAI's public release of ChatGPT. It is evidence that the race between Google/DeepMind and Microsoft/OpenAI is eroding ethics and safety.

Demis Hassabis, CEO of DeepMind, urged caution in his [recent interview](https://time.com/6246119/demis-hassabis-deepmind-interview/) in *Time*:

> He says AI is now “on the cusp” of being able to make tools that could be deeply damaging to human civilization, and urges his competitors to proceed with more caution than before.

> “When it comes to very powerful technologies—and obviously AI is going to be one of the most powerful ever—we need to be careful,” he says.

> “Not everybody is thinking about those things. It’s like experimentalists, many of whom don’t realize they’re holding dangerous material.”

> Worse still, Hassabis points out, we are the guinea pigs.

Alphabet/Google is trying to accelerate a technology that its own subsidiary says is powerful and dangerous.
# Meals For Unclear Dietary Restrictions

When someone's coming for over for dinner, I'll ask about their dietary restrictions. In my social groups it's common for people to be vegetarian or vegan, and others have a food allergy, are gluten-free, etc. Occasionally, however, I find myself making food without knowing my constraints. Perhaps someone's "bringing a friend" or when I asked they said "my food situation is pretty complicated, I can eat after if there's nothing that works for me." In these cases one option is to serve a meal that works for a wide variety of dietary restrictions.

I wouldn't want to only cook like this: every restriction you put on your cooking makes it harder. (Not specific to cooking!) Still, there are a few meals that do work well this way. For example, this is the main situation where I'll cook baked potatoes with stuff to put on them:

*   Baked potatoes
*   Black beans
*   Cheese sauce
*   Ground beef or bacon
*   Caramelized onions
*   Roasted cherry tomatoes
*   Chopped scallions
*   Olives
*   Sour cream
*   Baby spinach

This is mainstream American food, but because each person chooses what goes on their plate it works for nearly everyone: vegetarians leave out the meat, vegans leave out the cheese and cream, low-carb people leave out the potato, [potato-only people](https://www.lesswrong.com/posts/oh3YWqobjHAjNvNqd/potato-diet-a-post-mortem-and-an-answer-to-smtm-s-article) just eat the potato, there's no gluten, etc. Which exact sides you choose don't matter too much as long as you think a bit about dietary categories. I'm probably listing a few more than I would normally make.

If I didn't want to make potatoes I'd do tacos/burritos, without trying for something authentic. This could be:

*   Corn tortillas, briefly heated
*   Flour tortillas, same
*   Black beans
*   Grated cheese
*   Ground beef, chicken, or carnitas
*   Pico de gallo
*   Salsa
*   Roasted bell pepper slivers
*   Guacamole \[1\]
*   Sour cream
*   Baby spinach

And yes, this has a lot in common with the previous meal, but the spices are pretty different and I wouldn't mind serving them on consecutive days.

Other meals where there are a lot of options on the table and each person chooses their own combination also work. The main things I keep in mind are trying to keep each component relative simple, having at least two protein options, at least one vegan protein option, and at least one non-glutenous starch option.

  
\[1\] I usually don't make guacamole, because I don't really like it and I prefer not to prepare food I don't enjoy. It's not that I object to other people enjoying things when I don't, it's that [my cooking involves](https://www.jefftk.com/p/how-i-cook) a lot of tasting things and that strategy falls apart if you can't tell which direction is better. But it definitely belongs on this list if I'm not cooking, and even when I am a housemate will often make some.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0bhq8qZukyMj4aEsLu3xS7C4SpPoRnu5QVXVgGrs8nyrjx1b7sDktdEd7tWXR6KHVl), [mastodon](https://mastodon.mit.edu/@jefftk/109740268057193093)*
# A post-quantum theory of classical gravity?

This [paper](https://arxiv.org/pdf/1811.03116.pdf) proposes a unified theory of gravity and quantum physics which:

1.  Is empirically testable ([for example](https://arxiv.org/pdf/2203.01982.pdf))
2.  Gravity itself is non quantum
3.  physics is inherently stochastic (God does play dice)
4.  Objective waveform collapse exists ("the interaction  
    of the quantum degrees of freedom with classical space-time necessarily causes decoherence")

Are there any obvious reasons to dismiss this out of hand?

Would have lots of interesting implications (e.g. objective waveform collapse means no multiverse).
# What area of the digital domain seems safe from AI in the next 5-10 years?

I see a lot of digital services being built when they will quickly be automated by artificial intelligence. It's as if there's a disconnect between people who are aware of the rapid advances in AI and the average person. I see the same thing in education in my country, France. Students are preparing to go to university for a degree and skills that will be completely obsolete in 5 years. Even in computer science, everyone is promoting the idea that you have to learn to code to become a code worker, while automation tools are advancing at a rapid pace.  
As for content creators, the on-demand generation of text, video, and music could quickly make them irrelevant because most people copy other people. It's as if only people with real creativity will survive. I suspect that AI will tell us within 30 seconds that our innovative idea already exists on the Internet.  
I read on Twitter that the age of hackers is over and the age of people with ideas is beginning.  
I get lost every time I think about what will not soon be automated by AI in the digital domain.
# How to find cool things in a new place

A friend asked me for advice on how to find stuff that's going on in a place. I feel a bit self-conscious positioning myself as some kind of expert here, but I do think I've done a good job of this in Oxford. None of this is galaxy-brain stuff, just advice that I hope might be helpful for people.

The main trick is to talk to people. This is true, and bears repeating: talk to people. It helps a lot to cultivate a relentless curiousity, and to be comfortable being out of place (or at least, to let the discomfort wash over you). It's people who make these things happen, publicity etc is secondary.

Notice what you're interested in, what you enjoy, and do it even if it's kinda lame. Mention the mildly-embarrassing stuff you're doing, early in conversation. This will help people know what to recommend to you.

There's a saying, that if you wanna source drugs in a new place, ask your sketchiest friend who their sketchiest friend is, meet them, and repeat. This is true in general: if you wanna find a scene, talk to the people you know who are most in that direction, and ask them about their friends.

If you're at a place and it's good, ask the people there how they found out about it. Often, it's word-of-mouth, but sometimes there's a hub. In Oxford, the answer might be something like "a flier at the Magic Cafe" or "Instagram" or "Daily Info". Check these places, see what's there.

I don't use dating apps any more, but some of my most important let's-say-platonic friendships have come from that. Being bi is helpful, but whatever your situation I recommend matching/meeting people explicitly for friendship.

Perhaps the strongest move you can make is to host something. What do you most wish existed in your area, that doesn't seem to? If you can make this happen, maybe starting with a very small group, then good people will start to come to you. Hosting [Oxford Rationalish](https://www.lesswrong.com/groups/wQA8BE5e8mETeWb8A) has brought me in contact with so many people who I'd otherwise never have found.

Oh! And it's more important to take lots of shots than to have a perfect hit rate. I really like ending up in random places that don't make sense and that I probably wouldn't revisit, which exposes me to more upside.

Hoping to be helpful <3
# Quick thoughts on "scalable oversight" / "super-human feedback" research

The current default view seems to roughly be:

*   Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)
*   Scalable oversight is the only useful form of outer alignment research remaining.
*   We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.
*   But maybe it's not good?  Because it's more like capabilities research?  
     

A common example used for motivating scalable oversight is the "AI CEO".

My views are:

*   We should not be aiming to build AI CEOs
*   We should be aiming to robustly align AIs to perform "**simpler"** behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they *can* competently judge.
*   We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs).
*   From this PoV, scalable oversight does in fact look mostly like capabilities research.
*   However, scalable oversight research can still be justified because "If we don't, someone else will".  But this type of replaceability argument should always be treated with extreme caution.  The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms.  2) The details matter, and the tipping points are in different places for different types of research/applications, etc.
*   It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification").  For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless *both* processes conclude it is safe.
*   Getting AI systems to safely perform **simpler** behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well.
*   EtA: the main issue I have with scalable oversight is less that it is advancing capabilities, *per se*, and more that it seems to raise a "chicken-and-egg" problem, i.e. the arguments for safety/alignment end up being somewhat circular: "this system is safe because the system we used as an assistant was safe" (but I don't think we've solved the "build a safe assistant" part yet, i.e. we don't have the base case for the induction).