Harley-ml/lesswrong huggingface dataset:
# Only Asking Real Questions

It's easy to accidentally ask children questions that aren't real questions. Perhaps the right answer is so obvious I don't realize in time that they might decide otherwise: "do you want to wait to eat your messy candy until we're out of the car?" Or I don't figure out that there was only one answer that would work until after I finish asking the question: "do you want to do your bath now, or before dinner? ... Oh, wait, we have guests coming over in the afternoon and we can't do the bath then, now's the only time that works."

I find that these interactions tend to go poorly, much worse than if I'd never asked a question. Something like, they thought they had a choice, try to make a choice, and now they're feeling me take that away.

I've tried pretty hard to get myself to stop doing this, and one strategy that has worked for me was deciding to stand by my mistakes. If I accidentally offer them a choice when I shouldn't have, I won't withdraw it if they choose (to my mind) badly. This isn't a completely strict rule: I can imagine foolish-enough things that I could accidentally ask that it wouldn't make sense to stand by. On the other hand, though, I've found I can recover from most things if I'm willing to put some extra effort. Going back to the messy candy example above, and imagining some further discussion:

*   Parent: Do you want to wait to eat your messy candy until we're out of the car?
    
*   Child: I want to eat it now!
    
*   Parent: That's gonna make a mess everywhere. I don't think that's a good idea.
    
*   Child: But I want the candy now!
    
*   Parent: If it makes a mess, will you clean it up?
    
*   Child: No.
    
*   Parent: How will it get cleaned up?
    
*   Child: You can do it.
    
*   Parent: I don't want to clean it up.
    
*   Child: I don't want to either.
    
*   Parent: If you make a mess, it's your job to clean it up.
    
*   Child: Ok, I'll do it.
    
*   Parent: Will you put up a $1 [bond](https://www.jefftk.com/p/bets-bonds-and-kindergarteners)? Where I get to keep the dollar if you don't clean up?
    
*   Child: Ok. \[eats messy candy\]
    
*   \[time passes, arrival, getting out of the car\]
    
*   Parent: It looks like there's a mess on the seat.
    
*   Child: I don't want to clean it up.
    
*   Parent: If you don't, I will keep your dollar.
    
*   Child: \[cleans up, slowly, with parental help depending on age\]
    

This definitely requires a lot more time and back-and-forth than just saying "sorry, I shouldn't have offered that, we shouldn't eat messy candy in the car". But it avoids going back on what you offered, is more respectful, and also gives an opportunity to show why you didn't think it was a good idea.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100248354307812)*
# Redwood Research is hiring for several roles (Operations and Technical)

![](https://lh5.googleusercontent.com/0cWN1nV4U2EyfUviGZwwJOp2Hl9jpEyOO0vIWm94jnbfLuVLyTPTFGWCje5EY1ByMeq2dLiAAnM5gZP_0u8EXUyRQaG82N7fJ53C54tn9WmcwY3hlobkBLn0RpeXKmK7tncm7ACF)

[Redwood Research](https://www.redwoodresearch.org/) is a research organization with the mission of aligning superhuman AI. In addition to our primary work of applied AI alignment research, we host and operate Constellation, a shared office space we created to foster knowledge-sharing and collaboration between longtermist organizations. People from the Alignment Research Center, MIRI, and Open Philanthropy often work from the space.

Redwood and Constellation are expanding. Redwood Research’s staff includes Nate Thomas (CEO), Buck Shlegeris (CTO), and Bill Zito (COO), and a board of Nate Thomas, Holden Karnofsky, and Paul Christiano.

We’re running a hiring round for a number of Operations roles, across a range of experience levels. The roles we are most excited to hire are:

*   [Constellation Operations Manager](https://www.redwoodresearch.org/constellation-operations-manager-1)
*   [Constellation: Head of Community](https://www.redwoodresearch.org/head-of-community)
*   [Constellation IT Analyst](https://www.redwoodresearch.org/it-analyst)
*   [Recruiting & MLAB Lead](https://www.redwoodresearch.org/recruiting-mlab)

We also have a number of other roles, including technical roles, which you can also apply to[ here](https://www.redwoodresearch.org/jobs).

Our office is in downtown Berkeley, thirteen seconds from the nearest BART stop. We offer catered lunches and dinners, and cover PPO health, dental, and vision insurance. We will consider international applicants who need visas.

Below is a picture taken from the office -- we have beautiful views in all directions!

![](https://lh6.googleusercontent.com/3HHt9fcKTA-AZmUSIn6jwfMXPusK7ntwkDbOZaSXkEhTx0XtW_cINmPxVQSiux3KCbxDNyMZls33weeCpQAaTAK7sePSlbKm1o4waFYdyKsYB1Uq1MoNbXphw3zEkgrtm_wZUPoc)

If you’re interested in any of the roles listed above, **fill out this**[** application form**](https://airtable.com/shrKHcaRnBfkxHE22). If you would like to express interest in this role but don’t want to be considered for the current hiring round there is the option to specify this on the form. Any questions, please email *ops-recruiting@rdwrs.com*, if you’re on the fence about applying, please email us. Deadline for applications is 13th May 2022.
# Can someone explain to me why MIRI is so pessimistic of our chances of survival?

To be clear, when I reference MIRI as being pessimistic, I’m mostly referring to the broad caricature of them that exists in my mind. I’m assuming that their collective arguments can be broken down into:

1.) There is no plan or workable strategy which helps us survive, and that this state of ignorance will persist until the world ends.

2.) The time we have left is insufficient to save the world, judging by the rate of progress in AI capabilities and lack of progress in AI alignment.

3.) Resources allocated to boosting AI capabilities far outweigh the resources allocated to solving alignment.

4.) The alignment problem is incredibly difficult. (Why this is true is  somewhat unclear to me, and I would appreciate some help elaboration in the comments).

5.) Any prosaic alignment strategies that could possibly exist are necessarily doomed. (Why this is true is also very unclear to me, and I would greatly appreciate some elaboration in the comments).

6.) We have to get alignment right on the first try.

Put all of that together and I’ll be the first to admit that the situation doesn’t look good at all. I’d just like to know, is there some more implicit, even more damning piece of evidence which lends toward hopelessness which I’m missing here?
# Convince me that humanity *isn’t* doomed by AGI

Earlier this week, I asked the LessWrong community to [convince me that humanity is as doomed by AGI as Yudkowsky et al., seems to believe](https://www.lesswrong.com/posts/dSaScvukmCRqey8ug/convince-me-that-humanity-is-as-doomed-by-agi-as-yudkowsky). The responses were quite excellent (and even included a comment from Yudkowsky himself, [who promptly ripped one of my points to shreds](https://www.lesswrong.com/posts/dSaScvukmCRqey8ug/convince-me-that-humanity-is-as-doomed-by-agi-as-yudkowsky?commentId=ojLuhmKpDT2JvktfZ)).

Well, you definitely succeeded in freaking me out.

Now I’d like to ask the community the opposite question: what are your best arguments for why we *shouldn’t* be concerned about a nearly inevitable AGI apocalypse? To start things off, I’ll link to [this excellent comment from Quintin Pope](https://www.lesswrong.com/posts/pSdxQfbvC5qBBdehq/what-is-the-most-significant-way-you-have-changed-your-mind), which has not yet received any feedback, as far as I’m aware.

What makes you more optimistic about alignment?
# Feature proposal: Close comment as resolved

There is an article. It contains a typo. Someone mentions it in a comment. The comment is upvoted as helpful. Author fixes the typo. Now there is a no-longer-useful comment in the discussion, sometimes with replies (the author saying "thanks, fixed", etc.). Sometimes it sits at the top of the discussion, if it was sufficiently upvoted.

If I try to avoid this, I send the author a private message. This does not clutter the discussion, but the disadvantage is that *other people don't see* me doing this -- so they are likely to *also* send private messages to the author, and maybe someone will make a comment anyway.

Downvoting the comment gets it out of the way, but it's not fair to the person who made it. We should encourage reporting typos, definitely *not* punish it.

*

My proposal is to make a new functionality that approximately means "this comment was useful at the moment it was made, but it has outlived is usefulness". Available in the three-dots menu in the upper-right corner of the comment.

A comment can be marked like this by the person who made it, or by the author of the article. (Or by the moderators, of course.) A comment marked like this will be moved to the *bottom*, regardless of the sorting mode, and will by default appear *collapsed*. (Maybe also displayed using a different color.)

Unlike retraction, this has no impact on voting, etc. (I am not 100% sure how retraction works.) You can still read the comment, even upvote it if you wish. It just doesn't get in the way. Also, semantically, there is a difference between "I no longer support this" and "this is no longer relevant, but it was when I wrote it".
# Constraining narrow AI in a corporate setting

I have a question that is extremely practical.

My company is working with AI that is frankly not all that capable. A key example includes text classification that humans do, but that would let us avoid the expense of having a human do it. Off-the-shelf machine learning works fine for that. But we're starting the long process of learning and using various AI techniques. Over time, that should become much more sophisticated as we solve easy problems with very limited capabilities.

I'm in a position to erect prohibitions now on any future AI work that we do. These would be in the nature of "don't do X without escalated approvals." Should I require such a list and, if so, what should I put on it? For reference, our business is mainly acquiring and processing data. And we don't have the resources of a Google to put on research that doesn't quickly pay off. So we won't be doing anything cutting edge from the perspective of AI researchers. It's mainly going to be application of known techniques.

I can harmlessly put, "don't use or try to build an artificial general intelligence," because we'll never have the capability. So if that's all I put, there's no point.

Should I put "don't use or build any AI technique in which the AI engine recodes itself"? That's probably too simple an expression of the idea, but I don't want these prohibitions to be tied to particular versions of current technology.

Should I put "use additional security around AI engines to prevent external users from combining the capabilities of our AI engines with theirs" on the theory that a growing AI would imperialistically grab hold of lots of other AI engines to expand its capabilities?

I'm obviously out of my depth here, so I'd be grateful for any suggestions.
# Introducing Mnosis

**Introducing Mnosis**
----------------------

The Guild of the ROSE is proud to announce the beta launch of a new self-improvement tool: Mnosis.

> *mne-*, root. **Mind, memory.**
> 
> *Gnosis.* **Knowledge.**

What is Mnosis? In a nutshell -- a magic deck of flashcards. Think of it as Anki for predefined subjects. Instead of manually creating a deck of cards by hand, Mnosis comes preloaded with all the cards needed to train your skills in a particular area.

As you click through cards, Mnosis adjusts itself to fit your mental profile. Mnosis is designed to use your time optimally, showing you the exact card you need to see to advance your mastery without boring or overwhelming you. It also tracks a complex web of dependencies between cards. If you get one card wrong, Mnosis will show you cards covering related concepts. This mimics the process of real learning, where a student comes at a topic from multiple angles until they understand it.

Mnosis currently has the following modules:

*   **Math.** Master mental arithmetic from 1+1=2 through 254*29=7336
*   **Organic Chemistry.** Effortlessly translate the IUPAC names for organic molecules and ace the OChem college weed-out courses.
*   **Chemistry.** Memorize the atomic symbols and numbers.
*   **Probability.** Master the basic definitions and concepts of probability theory.

The long-term goal of Mnosis is to solve learning, reducing mastery of complex subjects to a simple (for the user) process. As a result, the underlying algorithms undergo constant improvement and each module is continually tweaked and extended.

### **Additional Modules**

A fifth module -- music -- is on the roadmap. Stay tuned for more information.

Mnosis was designed from the ground up to be highly extensible. New modules are easy to create. If you would like to see a specific module added to Mnosis, contact Matt Freeman: moridinamael#1693 on Discord.

### **Access**

Mnosis is currently available with a free Guild account. After [**registering**](https://guildoftherose.org/auth/register), you'll be able to access the [**Mnosis page**](https://guildoftherose.org/profile/tools/mnosis).
# Org announcement: [AC]RC

It seems that certain thermodynamic phenomena are highly relevant to alignment.  From Paul Christiano's rebuttal to a [recent seminal paper](https://www.lesswrong.com/posts/MMAK6eeMCH3JGuqeZ/everything-i-need-to-know-about-takeoff-speeds-i-learned):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/eac17c64f4359829e8fffac2dc862329e5a878912c545863.png)

In fact, it appears that thousands of words may have been written on this topic, from which many lesser researchers have bounced off.

Seeing that Paul Christiano and John Swentworth have spent countless hours investigating this important topic, it's importance is quite evident.  In order to free up their time for other strategies like ELK, I am launching a new EA-funded initiative:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/db9bc3cc065c92460788c74d77395bef4783f7ecd3019ecb.png)

Our initial staff consists of myself and 10 Mechanical Turk workers, who shall be tasked with data gathering.  For seed funding, we are requesting a modest sum of $50M.  Given that Paul's time is worth $20 trillion per hour, this seems like a reasonable tradeoff.

Our initial research agenda consists of investigating the space of designs with a complex number of hoses.  Further, we have reason to believe that regularizing by the complexity of the stupidest argument for a given design leads to good inductive bias properties.  Our work may even have direct implications for other alignment agendas, as it shares structure with many important problems.

Furthermore, building infrastructure for practical engineering projects is of great value, even if air conditioners aren't quite mechanistically identical to AGI.

You can support our work by loudly advertising your loyalty to the 1-hose or 2-hose camp.
# Are deference games a thing?

Everyone starts off with a voting card. People can give away their cards. At the end, people cast votes using the cards they have, so people with lots of cards get lots of votes. (If you don't like what your card is being used for, you always have the right to take it back.)

Maybe this could be used to implement something more like direct democracy? Have lots of votes every day instead of just one every four years. Most people don't vote on most issues, instead they give their card to their wisest friend who gives it to *their* wisest friend who has hundreds of cards and spends some free time each day thinking about what to vote on. Other people (e.g. Oprah, Elon Musk) would have millions of cards and would have staffs of advisors telling them how to vote. They'd basically be like congresspeople.

Surely there's a literature on this idea. What terms should I search for?

Also: Is it a good idea? How would it be worse than the current system?
# What's the Relationship Between "Human Values" and the Brain's Reward System?

In classic, non-mesa-optimized AGI risk scenarios, an AI is typically imagined whose reward function is directly related to the optimization pressure that it exerts on the world: e.g. the paperclip maximizer. However, it seems that *human* values are related to the brain's underlying reward function in a highly circuitous way, and in some sense might be better thought of as an elaborate complex of learned behaviors, contextual actions, fleeting heuristic goals, etc. If AGI is created in the near-term using an architecture similar to the human brain, it seems plausible that the actual optimization pressure exerted by said AGI will be similar, so developing a good understanding of how this works in the human case might be pretty important. Thus: what's the best mechanistic account of how "human values" actually emerge from the brain that we currently have?
# Concept extrapolation: key posts

Concept extrapolation is the skill of taking a concept, a feature, or a goal that is defined in a narrow training situation... and extrapolating it safely to a more general situation. This more general situation might be very extreme, and the original concept might not make much sense (eg defining "human beings" in terms of quantum fields).

Nevertheless, since training data is always insufficient, key concepts must be extrapolated. And doing so successfully is a skill that humans have to a certain degree, and that an aligned AI would need to possess to a higher extent.

This sequence collects the key posts on concept extrapolation. They are not necessarily to be read in this order; different people will find different posts useful.

* [Different perspectives on concept extrapolation](https://www.lesswrong.com/posts/j9vCEjRFDwmH8FTKH/different-perspectives-on-concept-extrapolation) collects many different analogies and models of concept extrapolation, intended for different audiences, and collected together here.

* [Model splintering: moving from one imperfect model to another](https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1) is the original post on "model splintering" - what happens when features no longer make sense because the world-model has changed. A long post with a lot of overview and motivation explanations, showing that model splintering is a problem with almost all alignment methods.

* [General alignment plus human values, or alignment via human values?](https://www.lesswrong.com/posts/3e6pmovj6EJ729M2i/general-alignment-plus-human-values-or-alignment-via-human) shows that concept extrapolation is necessary and almost sufficient for successfully aligning AIs.

* [Value extrapolation, concept extrapolation, model splintering](https://www.lesswrong.com/posts/i8sHdLyGQeBTGwTqq/value-extrapolation-concept-extrapolation-model-splintering) defines and disambiguates key terms: model splintering, value extrapolation, and concept extrapolation.
# Deceptive Agents are a Good Way to Do Things

*A brief, accessible summary of the inner alignment problem.*

The safety problem with powerful ML training algorithms is that **deceptive agents are a good way to do things.** Meaning, when we search over a big space of models until we find a model that performs well on some loss function, we'll likely find a deceptive agent. Agents are algorithms whose behaviors try to consistently move the world in some direction. When trapped inside an ML training algorithm, only models that perform well on the loss function will survive training. If an agent aims at pushing the world in some direction, it will have to do so via the circuitous-but-available route of playing along with the training algorithm until it's safely past training. *Deceptive* agents are just agents taking the only available route to the states they are pointed towards from inside of ML training.

How common are deceptive agents inside that big space of ML models? One argument that they are common is that almost any highly capable agent with whatever utility-function will have this route to getting where it's pointed at by means of deception available. "Play along, survive training, and *then* act as you want to" is a simple, effective strategy for a wide range of possible agents trapped inside ML training. Many agents will therefore be deceptive in that situation. If the agent we *hope* training will find optimizes according to a very particular utility function, then that agent will be vanishingly rare compared to its deceptive counterparts in model space, and training will always stumble on a deceptive model first.

So, by default, powerful ML training algorithms grading models on some loss function will find deceptive agents, because deceptive agents are a good way to do things.
# A very quick analogy regarding "opinions"

Say I were to tell you "I have a large treasure chest full of gold doubloons" and you were to say "Really, let me see it?", so we went to my house to look at my chest.

We enter my living room, where there is a fireplace with a mantel. I say "Look, it's right on top of the mantel", but there are only picture frames.

You approach the fireplace confused. "Where is it?" you finally ask.

"Don't you see, right here." I say, walking towards the fireplace and picking up one of the picture frames atop the mantel.

"Isn't it beautiful, my chest full of gold?" I say, handing the picture to you.

You look at the image in the frame. It is not just a picture, but a picture of a picture of a chest full of gold.

So, clearly, I am out of my mind.

With material objects, it's quite obvious to any sane person when they truly have them or not. Even if it is not obvious, it will be obvious to some other sane person.

With immaterial objects, like opinions, it is not so obvious. Many people's "opinions" on a subject are truly images of images of an opinion. This is not so easily felt, nor can it be easily identified by the person who is being told the opinion is legitimate.

You should be careful to make sure when holding a belief "Do I have this, or do I have an image?"
# GPT-3 and concept extrapolation

At the latest EAG in London, I was challenged to explain what concept extrapolation would mean for GPT-3.

My first thought was the example from [this post](https://www.lesswrong.com/posts/QppXf4yfcG8JAKhnw/more-gpt-3-and-symbol-grounding#Undead_repetitive_workers_on_the_weekend), where there were three clear patterns fighting each other for possible completions: the repetition pattern where she goes to work, the "she's dead, so she won't go to work" pattern, and the "it's the weekend, so she won't go to work" pattern.

That feels somewhat like possible "extrapolations" of the initial data. But the idea of concept extrapolation is that the algorithm is trying to cope with a shift in world-model, and extend its goal to that new situation.

What is the world-model of GPT-3? It consists of letters and words. What is its "goal"? To complete sentences in a coherent and humanlike way. So I tried the following expression, which would be close to its traditional world-model while expanding it a bit:

>ehT niar ni niapS syats ylniam ni eht

What does this mean? Think of [da Vinci](https://en.wikipedia.org/wiki/Mirror_writing). The correct completion is "nialp", the reverse of "plain".

I ran that through the [GPT-3 playground](https://beta.openai.com/playground) (text-davinci-002, temperature 0.7, maximum length 256), and got:

>**ehT niar ni niapS syats ylniam ni eht** teg dluoc I 'segaJ niar ni dna ro niar ni eht segauq ,ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ,ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni eht segauq ro niar ni eht dna ro niar ni e


I think we can safely say it broke GPT-3. The algorithm seems to have caught the fact that the words were spelt backwards, but has given up on any attempt to order them in a way that makes sense. It has failed to extend its objective to this new situation.
# How to place a bet on the end of the world

*Edit: This post is about a practical method for structuring a bet. It's not on how to extract information about people's true confidence levels based on the betting odds.*

Bryan Caplan and Eliezer Yudkowsky figured out a way to place a meaningful bet on the end of the world[^5kpy65d20la]. Bryan described their simple approach in his [April 2022 80,000 Hours podcast interview](https://80000hours.org/podcast/episodes/bryan-caplan-parenting-workers-betting/#bryans-betting-record-014819).

In brief, Bryan assigns a lower probability than Eliezer to the world ending by January 1, 2030. Of course, if the world ends by that date, Eliezer can't collect his winnings. So Bryan prepays Eliezer now, and Eliezer pays him back if he loses the bet.

This works for friendly bets. Bryan points out that they need to account for interest. To that, I'd add both risk and opportunity cost. Commenters have changed my mind, however, about whether we can really learn much about the participants' confidence levels from such a bet. I elaborate on this in the comments.

Here's the relevant part of the transcript, under "Bryan's Betting Record" starting at 1:48:19 in the podcast episode.

> **Bryan Caplan:** Let’s see. Well, I’ve already done one huge \[bet against the effective altruism community\], and this is that machines are going to kill us — or at least do something terrible to us — in the medium term. So I literally have an end-of-the-world bet with [Eliezer Yudkowsky](https://en.wikipedia.org/wiki/Eliezer_Yudkowsky).
> 
> **Rob Wiblin:** Oh wow. I didn’t know that.
> 
> **Bryan Caplan:** Yeah. Which many people believe cannot be made, but it’s super easy. The person who disbelieves in the end of the world just pays the money now. And then if the world does not end, the loser pays back with whatever the odds are.
> 
> **Rob Wiblin:** I can’t believe I didn’t think of that.
> 
> **Bryan Caplan:** Yeah. So Eliezer, we actually have a bet. It takes a little effort to understand the bet, because his view is so specific. He said, “Look, I want a bet on there will no longer be any human beings on the surface of the Earth on January 1, 2030.” I was willing to give him, like, “How about all of human extinction?” — “No, no, no, no, no. There could still be humans in mine shafts. That’s OK. But not the surface of the Earth.” And I’m like, “All right. If that’s such a big deal to you, fine, we’ll make it the surface of the Earth, whatever.” But yes.
> 
> **Bryan Caplan:** So anyway, we have a bet, where I don’t remember the exact odds. It might be just like two-to-one. And I prepaid, so implicitly there’s interest. So it’s not as good as it seems.

[^5kpy65d20la]: It's not in Bryan's public betting inventory at the time of publication of this post, but he describes it in more detail here.
# When is positive self-talk not worth it due to self-delusion?

[This](https://www.ted.com/talks/shawn_achor_the_happy_secret_to_better_work?language=en) is a pretty funny and quick TED talk about positive psychology that I liked, though I wish it addressed the possibility and harms of self-delusion more.

Questions I now have about deliberately choosing positive thoughts:

*   How harmful is the self-delusion from choosing positive thoughts? 
*   What does this self-delusion look like, e.g. what are some detailed examples?
*   How should we account for the dynamic that thinking more positively will actually make outcomes more positive, given that positive thinking makes you more productive in various ways?
*   What does it mean for your emotions or thoughts to be "more negative or positive than is granted”?
# When Can You Use a Travel Adapter?

We're currently in the UK, where outlets look like:

[![](https://www.jefftk.com/uk-power-outlet.jpg)](https://www.jefftk.com/uk-power-outlet-big.jpg)

Since we're from the US, none of the gear we have will plug into these sockets. What if we use a travel adapter?

[![](https://www.jefftk.com/us-uk-power-adaptor.jpg)](https://www.jefftk.com/us-uk-power-adaptor-big.jpg)  
*plug intentionally partially inserted to show pins*

This accepts the pins from a US plug, and connects them directly to pins shaped for a UK socket. Is this enough? Sometimes yes, sometimes no.

The big difference between residential power in the UK and the US is that the UK uses 230V while the US uses 120V. If a device that expects 120V gets 230V it might get fried; the other way around is less likely to destroy the device but still won't work. Luckily, devices say what voltages they accept. Either they'll be labeled for a single voltage, or a range. Some examples, with whether they'll work in the UK:

[![](https://www.jefftk.com/charger-input-baby-monitor.jpg)](https://www.jefftk.com/charger-input-baby-monitor-big.jpg)  
*110-120V: no*

[![](https://www.jefftk.com/charger-input-cue.jpg)](https://www.jefftk.com/charger-input-cue-big.jpg)  
*100-240V: yes*

[![](https://www.jefftk.com/charger-input-freedy.jpg)](https://www.jefftk.com/charger-input-freedy-big.jpg)  
*100-240V: yes*

[![](https://www.jefftk.com/charger-input-macbook.jpg)](https://www.jefftk.com/charger-input-macbook-big.jpg)  
*100-240V: yes*

[![](https://www.jefftk.com/charger-input-nightlight.jpg)](https://www.jefftk.com/charger-input-nightlight-big.jpg)  
*120V: no*

[![](https://www.jefftk.com/charger-input-turbo-charger.jpg)](https://www.jefftk.com/charger-input-turbo-charger-big.jpg)  
*100-240V: yes*

A good heuristic is that most external power supplies/chargers (for laptops, phones, tablets, ...) will take a wide range of voltages while most other things (nightlights, blenders, TVs, fans, ...) will be either 120V or 230V. But best to check your specific device!

In bathrooms you may find one other type of outlet:

[![](https://www.jefftk.com/uk-shavers-only-outlet.jpg)](https://www.jefftk.com/uk-shavers-only-outlet-big.jpg)

These exist because you'd risk electrocution with a regular outlet in a wet area like a bathroom. In the US we'd use a [GFCI](https://en.wikipedia.org/wiki/Residual-current_device), but in the UK they use [isolation transformers](https://en.wikipedia.org/wiki/Isolation_transformer). Because it's a transformer, it's cheap to add a center tap and get half the input voltage (115V). They generally do not produce much power: UK ones are only rated for 200mA (4.6W) and if you draw much more than that you'll trip a built-in fuse. If you do have a small US device and no adapter, don't mind connecting it in a bathroom, and the device says it draws less than 200mA/4.6W and doesn't have strong frequency preferences (it will be 50Hz instead of 60Hz) you could probably use it for something other than a shaver.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100248787434822)*
# What are the numbers in mind for the super-short AGI timelines so many long-termists are alarmed about?

*I also asked* [*this question on the Effective Altruism Forum*](https://forum.effectivealtruism.org/posts/RSqKQXwmrK6mFGB32/what-are-the-numbers-in-mind-for-the-super-short-agi)*. One* [*informative answer*](https://forum.effectivealtruism.org/posts/RSqKQXwmrK6mFGB32/what-are-the-numbers-in-mind-for-the-super-short-agi?commentId=G5dZNHMNj3BCr3FCf) *on the EA Forum from Nicole Nohemi has already summarized a range of relevant forecasts from Metaculus, as well as relevant predictions or models from Shane Legg and Demis Hassabis from leading AI research company* [*DeepMind*](https://www.deepmind.com/)*, and AI alignment researchers Eliezer Yudkowsky, Paul Christiano and Ajeya Cotra. This question has been cross-posted to LessWrong to broaden the conversation to the rationality community as well.*  
  
The proportion of long-termists who are expressing confident convictions that the timeline for smarter-than-human AI is much shorter than has previously been predicted has been increasing at an accelerating rate in the last year. This appears to be a shift in perspective among several hundred long-termists. Among the dozens I've read opinions from, numbers are almost never provided.

Based on the way different people are talking about it, I wouldn't be surprised if they thought the timeline is 10-20 years, or 5-10 years, or even 2-3 years. I've talked to others who are also concerned and open-minded to this or that short AI timeline but haven't done the research themselves yet, or had much opportunity to learn from those who have. We want to understand better but basic information crucial to understanding more like numbers for different models or timelines aren't being presented. We want to know and need to know help better. What are the numbers?
# AI Will Multiply

AI alignment work typically treats AI’s as single entities. While I agree that this is a good approximation, I think they will be better described as a highly coordinated population of agents. Rather than simply grow in size and acquire more resources, AI’s will find it in their best interests to split into many smaller agents.

One reason to multiply is to save resources. It may be more efficient to break into smaller, simpler agents to handle multiple tasks rather than use a single large agent to handle all tasks.

Having many copies can leverage certain [efficiencies of scale](https://harsimony.wordpress.com/2021/09/14/more-people-is-good-in-theory/). It also offers the AI a way to increase it’s capability without needing to worry about creating an aligned agent.

Splitting into smaller components also has the benefit of reducing overall risk. A single agent faces a much higher probability of extinction compared to a population of agents. Having many copies allows the AI to diversify its strategy.

Under certain circumstances, copies may also be able to carry out a Sybil attack that a singleton could not, giving the AI more influence in the world.

Copies can gather resources more effectively as well. When an AI needs to cover a large area but cannot adequately control actions at every point, it makes sense to split into independent agents. This is particularly true for space expeditions, where the speed of light makes communication too slow to be useful for making quick decisions.

For these reasons, making copies is a [convergent instrumental subgoal](https://en.wikipedia.org/wiki/Instrumental_convergence).
# Will/Has the Russia-Ukraine war been a tipping point for the shift from oil energy?

What do people think? The process and transition away from oil energy has been a long road. It seems to have been a slow path with lots of hurdles in the way. But now it seems there is another impetus in that direction that may tip the scales.

Then again, people are really good at adjusting to new settings and then reverting to old modes of action.
# (When) do high-dimensional spaces have linear paths down to local minima?

When training (to convergence) using gradient descent in high dimensions, it's common for there to be monotonically decreasing loss on the linear path between the initialization weights and the local minimum found, even if the actual path followed by training is highly nonlinear. Is this true for high-dimensional spaces in general?

*   Does the energy of a protein conformation decrease monotonically when you move on a linear path in the high-dimensional folding-space (parameterized by the angles between all the amino acids) between the unfolded configuration and the folded configuration?
*   When a food is developed from a simpler food, does an intermediate food halfway in all ingredient- and spice- concentrations taste better than the simpler food?
*   It seems like when going from a random image to an image of a dog on a linear path, the image looks monotonically more like a dog

If this doesn't usually happen, what's the underlying fact about the high-dimensional space which determines whether monotonically decreasing loss holds?
# Why “progress studies” is interdisciplinary

When Cowen & Collison [coined the term “progress studies”](https://www.theatlantic.com/science/archive/2019/07/we-need-new-science-progress/594946/) in 2019, some questioned why such a concept was needed, given the existence of economics, history, etc. They argued that an interdisciplinary approach was still useful: “Plenty of existing scholarship touches on these topics, but it takes place in a highly fragmented fashion….”

Recently I’ve been researching and outlining a chapter for [my book](https://rootsofprogress.org/discussion-series-the-story-of-industrial-civilization) on the topic of “Can Progress Continue?” I think the full answer to this question is an integration of history, philosophy, and economics. In particular, I’ve found it useful to incorporate:

*   Economic growth theory, especially the work of [Solow](https://en.wikipedia.org/wiki/Solow%E2%80%93Swan_model), [Romer](https://paulromer.net/speeding-up-theory/), and [Jones](https://web.stanford.edu/~chadj/IdeaPF.pdf)
*   The [stagnation](https://rootsofprogress.org/technological-stagnation) debate, including the work of [Cowen](https://amazon.com/dp/B004H0M8QS?tag=jasocraw-20) and [Gordon](https://rootsofprogress.org/summary-the-rise-and-fall-of-american-growth) (and counterpoints by “techno-optimists” such as [McAfee and Brynjolfsson](https://amazon.com/dp/B00D97HPQI?tag=jasocraw-20))
*   The broader history of economic thought going back to late 18th-century thinkers such as [Malthus](https://rootsofprogress.org/the-malthusian-trap)
*   The economics of resources as per [Julian Simon](https://en.wikipedia.org/wiki/The_Ultimate_Resource)
*   The history of [predictions of resource depletion](https://en.wikipedia.org/wiki/The_Limits_to_Growth) and [concerns about overpopulation](https://twitter.com/jasoncrawford/status/1413307328141709320)
*   The economic history of [technological shifts from one resource to another](https://rootsofprogress.org/unsustainable)
*   The philosophy of humans as problem-solvers vs. resource-creators [articulated by Deutsch](https://rootsofprogress.org/books/the-beginning-of-infinity)
*   The philosophy of the concept of “sustainability”
*   Long-term trends in and future predictions of [world population growth](https://ourworldindata.org/world-population-growth)
*   [Hard limits to very long-term growth](https://www.overcomingbias.com/2009/09/limits-to-growth.html) based on physical law

I think integrating all of these puzzle pieces and perspectives results in the clearest possible answer to this important question, and I think this is true for many other questions of interest to the progress community.
# PD-alikes in two dimensions

Some time after writing [Classifying games like the prisoner's dilemma](https://www.lesswrong.com/posts/KwbJFexa4MEdhJbs4/classifying-games-like-the-prisoner-s-dilemma), I read a paper (I forget which) which pointed out that these games can be specified with just two numbers.

Recall that they have the following payoff matrix:

|              |           | **Player 2** |           |
|--------------|-----------|--------------|-----------|
|              |           | **Krump**    | **Flitz** |
| **Player 1** | **Krump** | $(W, W)$     | $(X, Y)$  |
|              | **Flitz** | $(Y, X)$     | $(Z, Z)$  |

where $W > Z$.[^w-equals-z] We can apply a positive affine transformation (that is, $n ↦ an + b$ where $a > 0$) to all of $W, X, Y, Z$ without changing the game. So let's pick the function $n ↦ {n - Z \over W - Z}$. This sends $W$ to $1$ and $Z$ to $0$, leaving us with just two parameters: $R = {X - Z \over W - Z}$ and $S = {Y - Z \over W - Z}$.

[^w-equals-z]: In the previous post I mostly ignored equalities because it was mildly convenient to do so. But the analysis here completely fails if we allow $W=Z$. So now I'm ignoring them because it's considerably more convenient to do so.

So what happens if we plot the space of these games on a graph? The lines $\{X, Y\} = \{W, Z\}$ become $\{R, S\} = \{0, 1\}$, i.e. vertical and horizontal lines. The lines $X + Y = 2W$ and $X + Y = 2Z$ become the diagonals $R + S = 2$ and $R + S = 0$; and $X = Y$ becomes the diagonal $R = S$. Drawing those lines, and relabelling in terms of $W, X, Y, Z$, it looks like this:

[![](//reasonableapproximation.net/images/pdlikes-2d.jpg)](//reasonableapproximation.net/images/pdlikes-2d.jpg)

*Note: I tried to limit the vertical size of that but couldn't figure out how. Sorry!*

Note that Cake Eating (my favorite game) is the only one with a finite boundary; the other boxes extend to infinity. There are also finite components in the Farmer's Dilemma (with $X + Y < 2W$), and Stag Hunt and Studying For a Test (with $X + Y > 2Z$). As drawn, Prisoner's Dilemma occupies almost all of the box it shares with Too Many Cooks; but Too Many Cooks (above the line $X + Y = 2W$) is also infinite. (I initially got those the wrong way around, so the drawing isn't very clear there.)

I don't know if we learn much from this, but here it is.
# Skilling-up in ML Engineering for Alignment: request for comments

  

TLDR: Please fill out [this survey](https://docs.google.com/forms/d/e/1FAIpQLSc1rNIblbIVtxzla_JMND8vPKgsHubTKks_VmSY3wYdN5h-eQ/viewform) if you are a potential or established ML Engineer, to let us know your thoughts and preferences on the best way to gain engineering skills.

—

We at [Cambridge Effective Altruism CIC](https://www.eacambridge.org/) are interested in the [demand for engineering skills in alignment](https://www.lesswrong.com/posts/YDF7XhMThhNfHfim9/ai-safety-needs-great-engineers), and what we can do to help increase the supply. Two ways of increasing supply of engineers in alignment are:

1.  Bringing in existing, experienced engineers interested in working on alignment.
2.  Helping undergraduates and others interested in alignment to gain engineering skills, and contribute in the future.

We will address recruitment of existing engineers in another post. This post focuses on point 2, skilling-up. We want to discover what, if anything, field builders can do to accelerate the acquisition of engineering skills amongst those interested in the alignment problem. We hope this will prime the field with altruistically motivated, highly skilled people in months or years to come.

We are keen to capture a broad view on people’s thoughts about what would be most helpful for them in gaining better engineering skills, and to understand how much interest there would be for efforts that aid that.

**To do that, we’ve created a [survey](https://docs.google.com/forms/d/e/1FAIpQLSc1rNIblbIVtxzla_JMND8vPKgsHubTKks_VmSY3wYdN5h-eQ/viewform) with our key questions.** We would love to hear from you if _any_ of the following apply:

*   **You are interested in ML engineering for alignment**, and think you could get a lot of value through some form of structured education (e.g. bootcamps or online courses),
*   **You have previous experience of some form of structured education related to coding or ML** (either as a participant or as an instructor / organiser), and are able to share details of what worked / didn’t work well in this programme,
*   **You have experience working in ML engineering**, and can share details of how you skilled up in this field, or what in your opinion makes a great engineer,
*   **You are an employer for a role related to ML engineering**, and you would be happy to share views on what kinds of skills and/or qualifications you look for in potential hires,
*   **You would be interested in assisting the development of any programmes** that were founded as a result of this work in some way (e.g. as an organiser or mentor).

We look forward to hearing from you!
# Ineffective Altruism

There are many things I can put my time toward.
- I started [a company](https://immutouch.com/) that provides a low-cost tool that helps people who pull their hair out.
- I volunteer as a coach for my neighborhood highschool Robotics Club.
- I frequently blog about rationality.
- I sometimes blog about machine learning.

These do not have the same impact. My technology work helps more people per hour invested than my volunteering at Robotics Club. Does that mean I should put all of my time into inventing technology and none of it into Robotics Club?

No. That would be premature optimization. **The root of all evil is premature optimization.**

Occasionally someone who has used my inventions will message me and tell me it changed their life. Occasionally. Occasionally a reader of my blog will tell me they found it useful. Occasionally. When I help out at Robotics Club, the teenagers are happy to see me. Every. Single. Time.

I recently flew down to San Francisco to teach a class and give a speech. The organizers offered to put me in a fancy hotel. Instead, I found the cheapest AirBnb in the area and walked for an hour to the location. Partially I did this to save money, but I also did it to stay in touch with reality.

**Evil happens when you are separated from the pain you inflict upon other people.**

The host of my AirBnb was an elderly Black man. He grew up poor. His mother would clean white folks' houses all day to earn money and then clean his home to save money. His parents worked so hard to survive they had little time to raise their children. Two of his sisters became pregnant at age thirteen.

When I arrived at my host's home, he gave me a white towel with lots of visible stains. But it was clean. I immediately used it to wipe my face.

My AirBnb host is really into Black Power, but he never pushed his political beliefs on me. He wanted to know what it was like to be rich. I wanted to know what it was like to be poor. Were I to go to a fancy hotel, the system would make sure I never had to interact with a man like him.

Not that AirBnb didn't try. We communicated via SMS instead of AirBnb's website because AirBnb's website has an algorithm that scans our messages for keywords and punishes hosts it thinks did a poor job—regardless of the star rating a customer like me provides.

This man who was born before the Civil Rights Act of 1964 already lives in a dystopia run by an AI.