Harley-ml/lesswrong huggingface dataset:
# Built-in Measuring Spoons

In making the kids breakfast each morning, which has moved on from [crepes](https://www.jefftk.com/p/eggy-crepes) and [waffles](https://www.jefftk.com/p/eggy-waffles) to pancakes, I want a consistent amount of baking powder. I realized that the container has a built-in "measuring spoon". If you shake it with the lid on, powder gets up above the leveling shelf:

[![](https://www.jefftk.com/baking-powder-lid-on.jpg)](https://www.jefftk.com/baking-powder-lid-on-big.jpg)

Take the lid off, and it remains:

[![](https://www.jefftk.com/baking-powder-lid-off.jpg)](https://www.jefftk.com/baking-powder-lid-off-big.jpg)

Then you can dump it in the batter:

[![](https://www.jefftk.com/baking-powder-dumped.jpg)](https://www.jefftk.com/baking-powder-dumped-big.jpg)

It's a bit awkward to do this with a full container, since you can't tip it very much, but it's quite easy once you've used a bit first.

With the particular brand of baking powder we've been buying it's consistently 2/3 of a teaspoon.

I wonder whether anyone has tried designing a baking powder container to make this easier? The current shape is not ideal for measuring, and it would make much more sense for it to be one teaspoon instead of 2/3.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100236473502042)*
# $100/$50 rewards for good references

*With thanks to Rohin Shah.*

Dear LessWrongers, this is an opportunity to make money and help with AI alignment.

We're looking for specific AI capabilities; has anyone published on the following subject:

* **Generating multiple reward functions or policies from the same set of challenges**. Has there been designs for deep learning or similar, in which the agent produces multiple independent reward functions (or policies) to explain the same reward function or behaviour?

For example, in [CoinRun](https://openai.com/blog/quantifying-generalization-in-reinforcement-learning/), the agent must get to the end of the level, on the right, to collect the coin. It only gets the reward for collecting the coin.

![](https://www.dropbox.com/s/e5ggon1o9yebb2m/coinrun.png?raw=1)

That is the "true" reward, but, since the coin is all the way to the right, as far as the agent knows, "go to the far right of the level" [could just as well have been the true reward](https://arxiv.org/pdf/2105.14111.pdf).

We'd want some design that generated both these reward functions (and, in general, generated multiple reward functions when there are several independent candidates). Alternatively, they might generate two independent policies - we could test these by putting the coin in the middle of the level and seeing what the agent decided to do.

We're not interested in a Bayesian approach that lists a bunch of reward functions and then updates to include just those two (that's trivially easy to do). Nor are we interested in an IRL-style approach that lists "features", including the coin and the right hand side.

What we'd want is some neural-net style design that generates the coin reward and the move-right reward just from the game data, without any previous knowledge of the setting.

So, does anyone know any references for that kind of work?

[We will pay](https://buildaligned.ai/) $50 for the first relevant reference submitted, and $100 for the best reference.

Thanks!
# Should we postpone getting a booster due to Omicron, till there are Omicron-specific boosters?

Wai Dai writes over in the comments at [Omicron Variant Post #1: We’re F***ed, It’s Never Over](https://www.lesswrong.com/posts/6apSCHHuWyxK635pE/omicron-variant-post-1-we-re-f-ed-it-s-never-over?commentId=MbvSmohprgXaGCDud):

> From your linked paper [https://www.cell.com/trends/immunology/fulltext/S1471-4906(21)00177-0:](https://www.cell.com/trends/immunology/fulltext/S1471-4906(21)00177-0) *\[Wai Dai's quote from the paper in italics\] *
> 
> *Experience from the IAV field indicates that multiple serial exposures to variant IAV strains might imprint (focus) the immune response to a narrow array of well-conserved viral epitopes relative to emerging subsequent strains and this might eventually reduce vaccine effectiveness. Randomizing multiple cohorts over time to either serial or partial immunization might also yield important neutralization data, although powering such studies for efficacy would be challenging. Larger observational studies of cohorts receiving serial immunizations might therefore also prove useful.*
> 
> This suggests that it may be a good idea to avoid getting a booster now (with the ancestral spike protein) and wait for an Omicron-specific booster instead. (Use other precautions to avoid infection in the meantime.)

Given that this is an important question for many of us, I consider it useful to have the conversation on the top level instead of only in the comments of the other thread.
# What have your romantic experiences with non-EAs/non-Rationalists been like?

Epistemic status: trawlin' for anecdotes/commiserations; satiating-morbid-curiosity

I'm also interested in non-romantic relationships, but I'm *particularly* interested in hearing about long term relationships between folks within EA and/or LW and someone[s] outside it. 

By outside of EA/Rat I don't just mean "not active in the communities". I mean: doesn't have the philosophical machinery of rationality [ethics/meta-ethics/epistemology/cog sci/etc] and doesn't seem inclined to pick it up. 

Things that I'm particularly curious about: 
- how long these relationships lasted
- how you feel about the relationship: in what ways was/is it successful or unsuccessful?
- if your partner doesn't have a "rationalist frame", how you deal with that
- any other random thoughts this elicits
# Scott Alexander's "Ivermectin: Much More Than You Wanted To Know"

This is a linkpost, created for the 2021 Review.

> I know I’m two months late here. Everyone’s already made up their mind and moved on to other things.
> 
> But here’s my pitch: this is one of the most carefully-pored-over scientific issues of our time. Dozens of teams published studies saying ivermectin definitely worked. Then most scientists concluded it didn’t. What a great opportunity to exercise our study-analyzing muscles! To learn stuff about how science works which we can then apply to less well-traveled terrain! Sure, you read the articles saying that experts had concluded the studies were wrong. But did you really develop a gears-level understanding of what was going on? That’s what we have a chance to get here!
# Are limited-horizon agents a good heuristic for the off-switch problem?

(This is my first post, sorry if this is covered elsewhere.)

Implicit in the problem of a superhuman AI's reward being misspecified and turning everything into paperclips is the fact that the agent is optimizing over e.g. "number of paperclips" without any particular time-bound, area-of-effect bound, or probability confidence bound. For example, imagine if a MuZero+++++ agent were given the reward function "maximize the expected amount of money in this bank account until 1 day from now, then maximize the probability of shutting yourself off", where e.g. "1 day from now" was set by a consensus of satellites and/or deep-space probes. The agent could do a lot of bad things via its misspecified reward, but at least the blast-radius seems like it'd probably be limited. This agent might cause a lot of damage, but it probably wouldn't end the world in pursuit of these objectives.

This is not a strategy for executing a Yudkowskian "pivotal act", which by necessity would need to have longer-lasting effects. So it wouldn't stop some other group from creating unfriendly AGI. But as a general AI safety heuristic to try and spread across AGI practitioners, "do this as much as possible for a fixed time and then shutoff" seems like a promising social intervention. We also *know how to implement it today. *

As far as specific risks from the pursuit of convergent instrumental goals: "capability gain" and "make sure you don't get shutoff" seem much less likely to be convergent instrumental goals when you are planning over a short fixed horizon.  "Create hidden copies of yourself" is still possibly a deployable strategy, but there's no reason for those copies to pursue a reward beyond the time-bound described, so I'd hold out hope for us to find a patch. "Deception" is again possible in the short term, but given this reward function there's no clear reason to deceive beyond a fixed horizon.

More broadly, this is a result of my thinking about AI safety social heuristics/memes that could be spreadable/enforceable by centralized power structures (e.g. governments, companies, militaries). If others have thoughts about similar heuristics, I'd be very interested to hear them.

I'm assuming I'm not the first person to bring this up, so I'm wondering whether someone can point me to existing discussion on this sort of fixed-window reward. If it is novel in any sense, feedback extremely welcome. This is my first contribution to this community, so please be gentle but also direct.
# Leaving Orbit

Duncan says:

> At CFAR workshops, we often installed this as an explicit local norm: "If at any point you want to convey 'hey, I'm bowing out of this conversation now, but it's not intended to be any sort of reflection on you or the topic, I'm not making a statement, I'm just doing what's good for me and that's all,' you just say '**Leaving orbit!**' and people will wave to you and keep on talking."

This strikes me as good social tech to try out on LessWrong (linking to this page to define the term).

Basic rationale:

1.  It's important to be able to randomly exit conversations. Otherwise, people won't add as much useful stuff to conversations in the first place (lest they be trapped).
2.  Sometimes people want to leave a message to indicate they're exiting. But it can be tricky to phrase such messages without implying one or more of: 
    *   "I think my last message is the last word in this conversation, and has resolved all outstanding issues."
    *   "I think you have raised no important or worth-responding-to points since my last message."
    *   "I think that you as a person are unworthy of my time."
    *   "Your last messages convinced me; I agree with you."
    *   "Your last messages didn't convince me; I disagree."
    *   "I have no idea how to object-level-respond to your messages."
3.  (Maybe some of these are things you *want* to communicate in some contexts, but it's good to not *have* to communicate them!)
4.  So having a very generic placeholder thing to say here is useful. "(Leaving orbit.)" seems friendly and generic enough to suit the bill.

* * *

Caveat: I think it's also 100% fine to exit conversations without leaving a message at all.

I generally think society has far too many "do this or you're bad" norms, especially [automatic norms](https://www.overcomingbias.com/2017/12/10-implications-of-automatic-norms.html). A bad outcome would be if "leaving orbit" (or similar) came to be seen as an *expectation* or *requirement*, rather than as an *option*. If this idea catches on *too* much, I suggest deliberately exiting more conversations without leaving any message, so as to keep this option from ossifying into an expectation. 🙂
# Retail Investor Advantages

There are some advantages to being a retail investor that might make it possible for you to beat the market without violating EMH.

Information Acquisition Cost
----------------------------

If you are an expert in some domain, it costs less for you to assess information in that domain than some trading firm. The trading firm has to spend money and suffer a time delay hiring an expert, incur the costs of communication overhead between people being difficult, also spend money on the opportunity cost of context switches and taking time away from traders at the company etc. Meanwhile, because you have both the trader and expert in your head, you save on a huge amount of overhead. Similarly, trading firms could [pay people to camp outside Walmart and count customers](https://www.lesswrong.com/posts/utySCY9nJt9xGYGGQ/the-emh-aten-t-dead#2__Hedge_funds_with_armies_of_drones), but if you already work at Walmart you can observe these things for relatively little marginal cost. Because of this, there are many small niches where the possible reward is too small to make it worth it for a trading company, but whose stakes are just the right size for retail investors. 

Cheaper Liquidity
-----------------

Nobody wants to be on the other side of the trade with RenTech, because RenTech makes smart trades. Everyone wants to be on the other side of the trade with retail investors, because most are uninformed and generally lose money over time. This is what makes it possible for Robinhood to offer commission free trades; uninformed flow is something that market makers really want to be on the other side of. The magnitude of trading fees often make or break a trading strategy.

Less Slippage
-------------

Moving less money moves the market less. As you scale up your Assets Under Management, your moves eat deeper into the order book and cause the market to move with you, eroding your profits. Large funds have entire teams of people dedicated to trade execution strategies to minimize their impact on the market. As a retail investor, you don't have this problem to the same magnitude; buying $100k of SPX will move the market a negligible amount. The magnitude of slippage also often makes or breaks a trading strategy; many naive backtests ignore slippage (because historical order book information is typically more difficult to obtain than OHLCV), and end up unprofitable in the real world because of it.

*(Nothing I post is financial advice, I am not a financial advisor, etc)*
# Interpreting the Biobot Spike

Boston has a really interesting program where it [measures covid RNA frequency during sewage treatment](https://www.mwra.com/biobot/biobotdata.htm). It's been running since the beginning of the pandemic:

[![](https://www.jefftk.com/ma-covid-biobot-2021-12-08.png)](https://www.jefftk.com/ma-covid-biobot-2021-12-08-big.png)

One thing I really like about this as a source of data is that it avoids some of the problems with tracking tests. It has no dependency on the fraction of people who decide to get tested, the reasons people decide to get tested, or the timing of tests. Sometimes they are slow in getting out updated numbers, but they always know which day a sample is from.

They publish the full data ([pdf](https://www.mwra.com/biobot/MWRAData20211207-data.pdf)), and I had to go at making a chart that's a bit less busy:

[![](https://www.jefftk.com/ma-wastewater-covid-biobot-2021-12-06.png)](https://www.jefftk.com/ma-wastewater-covid-biobot-2021-12-06-big.png)

As I wrote [on Saturday](https://www.jefftk.com/p/covid-christmas), we are seeing a serious spike right now, and since then we have one more day of data showing that it's continuing.

In his [most recent post on Omicron](https://thezvi.wordpress.com/2021/12/06/omicron-post-4/), [Zvi](https://thezvi.wordpress.com/) wrote:

> That spike on the right has two of the three highest single-day measurements, and they were the last two days of data reported. This can't represent Delta cases alone unless it's a data error, because the rise is too rapid given what we know about conditions. If it's Omicron...

I don't think this can be Omicron yet, because MA is doing variant testing, and as of Saturday detecting Omicron was still [newsworthy](https://www.mass.gov/news/omicron-variant-detected-in-massachusetts). To see whether this can be regular Delta, let's compare to last year:

[![](https://www.jefftk.com/biobot-2020-vs-2021.png)](https://www.jefftk.com/biobot-2020-vs-2021-big.png)

This is a similar pattern from fall 2020, just scaled up a bit. In the last month (November 6th to December 6th) we had 4.4x growth in 2020, and 4.9x in 2021. In the last week (November 29 to December 6th) we had 1.6x growth in 2020 and 2.1x in 2021. When you consider that in fall 2020 we didn't even have Alpha yet, let alone Delta, this seems like it is probably entirely coming from the time of year (seasonal change plus Thanksgiving).

On the positive side, however, at this time last year it leveled off for the winter aside from a short post-holiday spike, and if we're lucky we'll see the same this year.
# [Linkpost] Chinese government's guidelines on AI

China (PRC) published a new AI policy: "*Ethical Norms for the New Generation Artificial Intelligence*" (2021-09-26). 

A surprisingly large part of the document's content is dedicated to the topics related to AI X-risk (although the existential risk is never addressed directly). 

According to the document, it regulates AI research conducted by pretty much any entity in China, from companies to natural persons. 

The original official document in Chinese is [here](http://www.most.gov.cn/kjbgz/202109/t20210926_177063.html) ([archive](https://web.archive.org/web/20211030205120/http://www.most.gov.cn/kjbgz/202109/t20210926_177063.html)). Other AI policies are available [here](https://oecd.ai/en/dashboards/policy-initiatives?conceptUris=http:%2F%2Fkim.oecd.org%2FTaxonomy%2FGeographicalAreas%23China).

Some excerpts that could be relevant to AI X-risk: 

> AI activities should... follow the common values of mankind, respect... the fundamental interests of mankind,... promote human-machine harmony and friendliness. 

> Ensure that humans have full autonomy in decision-making,... the right to suspend the operation of artificial intelligence systems at any time,... ensure that artificial intelligence is always under human control. 
> 
> Actively learn and popularize knowledge of AI ethics
> 
> objectively understand ethical issues, and not underestimate or exaggerate ethical risks

> take the initiative to integrate AI ethics into the whole process of management

> Strengthen risk prevention. Enhance bottom-line thinking and risk awareness, strengthen the study and judgment of potential risks of artificial intelligence development, carry out timely and systematic risk monitoring and assessment, establish an effective risk warning mechanism, and enhance the ability to control and dispose of ethical risks of artificial intelligence.

> Strengthen the self-restraint of AI R&D-related activities, take the initiative to integrate AI ethics into all aspects of technology R&D, consciously carry out self-censorship, strengthen self-management, and not engage in AI R&D that violates ethics and morality.

> In the algorithm design, implementation, application and other links, enhance transparency, interpretability, understandability, reliability, controllability

> achieve verifiable, auditable, supervisable, traceable, predictable and trustworthy \[AI\].

> Prohibit the use of artificial intelligence products.... that endanger national security, public safety
# EA Dinner Covid Logistics

Yesterday we hosted an [EA](https://en.wikipedia.org/wiki/Effective_altruism) dinner, the first largeish (~22 people) gathering inside our house since before the pandemic. It went well, and it was really good to see people. I wanted to write a bit about what sort of precautions we took for covid.

[![](https://www.jefftk.com/ea-dinner-group-2021-12-11.jpg)](https://www.jefftk.com/ea-dinner-group-2021-12-11-big.jpg)

*   Open window in the main area with a fan pointing out. Windows partly open in other parts of the house, trying to balance the airflow so as much air as possible traveled past the guests. I preheated the house earlier in the day, and during the event had the heat on maximum. It was a bit chilly near the end, but still tolerable.
    
*   Air purifier [cubes](https://cleanarair.com/) in each room.
    
    [![](https://www.jefftk.com/air-filter-cube.jpg)](https://www.jefftk.com/air-filter-cube-big.jpg)
    
    In the main area, which isn't a good fit for one, I made a bit of a tunnel above the [bookcase](https://www.jefftk.com/p/built-in-shelves).
    
    [![](https://www.jefftk.com/filter-line-above-bookcase.jpg)](https://www.jefftk.com/filter-line-above-bookcase-big.jpg)
    
*   Masks on, except when eating. People were good about this; I didn't see people hanging out with mask off and a plate but not eating.
    
*   Rapid testing at the door ([$7/person](https://www.walmart.com/ip/BinaxNOW-COVID-19-Antigen-Self-Test-2-Count/142089281)). We lined up a volunteer to handle this, because Julia was dealing with hosting and I was watching the kids upstairs. With each test they wrote down the person's name, the time where it needed to be read and their cellphone number. When each test had been sitting for 15 minutes, we texted the person the result. We asked people not to take their mask off to eat until they had their results. If any tests had been positive, we would have informed the group. False positive rate is [~0.05%](https://journals.asm.org/doi/10.1128/JCM.00083-21). We are keeping the cell phone numbers for a few days, so we will be able to text everyone if anyone later notifies us they have it.
    
    [![](https://www.jefftk.com/ea-dinner-binax.jpg)](https://www.jefftk.com/ea-dinner-binax-big.jpg)
    
*   We told people vaccination was required, though not boosters. We intended to check vaccination cards (accepting photos) at the door, but forgot. If I was organizing some thing like this a few weeks from now, when there has been more time for people to get boosters, I might require them.
    
*   We considered requiring masks to be surgical or better (offering free ones) but decided not to. I think this probably would have been worth doing, except that we added several of these precautions after announcing the gathering, and it felt like too much to add another one.
    
*   Occasional crowd management, encouraging people to move from denser spaces to emptier ones
# Nuclear war anthropics

In the most recent episode of Rationally Speaking ("Humanity on the precipice", 10 Dec. 2021), Julia Galef and Toby Ord discuss the risk of a nuclear exchange during the Cold War:

> J.G. \[55:28\]: "If you had asked me to put a probability on each case, like taking nuclear near misses—for example, like the Cuban Missile Crisis or the time the Russians could have retaliated against us if not for Arkhipov—I would have put a probability of maybe 25% on a lot of those. But it starts to add up. If you take all of the near misses that I would've assigned 25% to, they add up enough that it starts to look really surprising that we didn't end up with a nuclear war, which kind of throws into question my ability to assign good probabilities to all of these near misses."
> 
> T.O. \[55:02\]: "You're right that if your attempts to do this are producing very large numbers, then that does suggest that the attempts might be going wrong, and I think it's very easy to have them go wrong; I think it's an extremely difficult thing, and I think there's been very little written, really, about how to do this kind of retrospective prediction or trying to assign probabilities to past events where we actually know what happened.

If anthropic effects were distorting the historical record, this is exactly what you'd expect to see: Julia's probabilities on nuclear exchange would seem reasonable individually but too high in aggregate. Why does neither of them mention the anthropics explanation? Is it somehow common knowledge that this explanation doesn't hold water?

*The Precipice* has a brief section on anthropics (part of Chapter 7), in which Toby says, "From what we know, it doesn't look like \[anthropic\] selection effects have distorted the historical record much, but there are only a handful of papers on the topic and some of the methodological issues have yet to be resolved." The only nearby citation is [this paper](https://www.nickbostrom.com/papers/anthropicshadow.pdf) by Cirkovic, Sandberg & Bostrom, who seem to conclude that selection effects from anthropogenic hazards are difficult to determine; they don't make any argument that those effects are small.

Also related: [https://www.lesswrong.com/posts/EkmeEAB646Yf2DNzW/anthropics-doesn-t-explain-why-the-cold-war-stayed-cold](https://www.lesswrong.com/posts/EkmeEAB646Yf2DNzW/anthropics-doesn-t-explain-why-the-cold-war-stayed-cold)
# Teaser: Hard-coding Transformer Models

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

In this post, I am trying to gauge potential community interest in a strand of research that I have been doing in my spare time off and on for the past year and a half (roughly).

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.
# Baby Nicknames

Lily and Anna rarely call Nora by her name. Instead, it's always nicknames, with lots of variety. For a week starting on Thanksgiving I counted how often they used different terms:

| 21  | Norkit (Anna favorite) |
| --- | --- |
| 20  | Nwetzel (Lily favorite) |
| 16  | Norky |
| 9   | Cuddleezel |
| 7   | Norka |
| 7   | Nori |
| 4   | Norit |
| 4   | Cuddleitz (former favorite) |
| 3   | Norweasel |
| 3   | Norklet |
| 2   | Norkla |
| 2   | Cuddlet |

Terms I only recorded once: Cuddleez, Cuddlehat, Cuddleoo, NW, Norahats, [Noranoo](https://www.jefftk.com/p/baby-sister-numbers) (former favorite), Norba, Norcuddly, Norfi, Norfkit, Norflea, Norfo, Norfu, Norito, Norkakit, Norkaloo, Norkanoo, Norkita, Norkito, Norko, Norkwinza, Norkwitz, Norquo.

I counted each term once per "session", where if they said "Norkit Norkit Notkit" I would still only count that once, while if an hour later they said "market" again I would give it another tally. Doesn't include any sessions where I had my hands full or otherwise wasn't able to easily tally.

It took me a while to get around to posting this, during which both kids, but especially Lily, have started using "Kaia" a lot. Asking Lily, she says it is "short for Cuddler".

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100237427180862)*
# Solving Interpretability Week 

For original motivation, see [solving corrigibility week](https://www.lesswrong.com/posts/Lv3emECEjkCSHG7L7/solve-corrigibility-week). I'll state what I learned from corrigibility week, why I didn't post one last week, and the updated format for interpretability. 

What I Noticed
--------------

The open call to co-work with my calendly link didn't work. Only met with people when I messaged them specifically. I'm changing [my availability](https://calendly.com/elriggs/chat?back=1&month=2021-12) from "all day Saturday" to "two hours/day" to better meet other's schedule and not have a single point of failure (see next section). It also seems great to ask specific people to meet when I have a question on their research.

Going through previous work, it was good to write out my thoughts in the google doc, spruce it up, and make it a comment on the original post. This also connects with messaging people to meet with if I feel there's a large disconnect or I have a lot of confusion regarding their work.

The google doc was also less long-term collaborative, which may have been from a lack of notifications that someone responded to what you wrote. So I'm moving the research direction part to the comment section here.

No Post Last Week
-----------------

I committed to posting 3 of these this month, but didn't last week. I had a bad Saturday (which was my designate work day for the corrigibility post) and a bad week until I talked to Shay, my therapist, and worked out a plan (Shay is great and has [free sessions for those working on alignment](https://www.aisafetysupport.org/resources/shay); highly recommend). I am not ashamed or embarrassed, but I do put less stock in my public commitments and more aware of my failure modes. I personally still recommend people try (and possibly fail) ambitious projects.

I do want to write a post on corrigibility, but it's too time-consuming to both work out my own thoughts going through the literature and meet with people to understand their work & distill those conversations. Both are important and not mutually exclusive. Some possible solutions are to circle back to corrigibility next year or take two weeks per topic. 

Format This Week
----------------

[Here is the google doc](https://docs.google.com/document/d/1mNPBQFO0Mp9GZvFWxuQL0yalIIg9Z_2p6xiGGHGfFm4/edit?usp=sharing) for interpretability. In the comments are top-level comments for:

1.  Research directions you want discussed with any questions you have
2.  Your meeting schedule for co-working and how you'd like to be contacted
3.  Suggestions for future weeks or changes to the format

In the doc are:

1.  Literature Review
2.  Tasks to do for further research

Again, in the google doc, it is socially acceptable to write low-quality babble. In this post's comments, I also accept babbling/ spit-balling and will not delete them.
# Language Model Alignment Research Internships

I'm [Ethan Perez](https://ethanperez.net/), a final year PhD student at NYU, working on aligning language models with human preferences. I'm looking to hire research interns to work on projects in this space, starting early 2022. I expect candidates to have strong software engineering ability, for ML engineering (e.g., to finetune GPT2 to good performance on a new task) or data engineering (e.g., to quickly find high quality subsets of text within petabytes of Common Crawl data). Ideal candidates will have experience doing ML and/or NLP research, reading papers, and coming up with ideas to test. I'm looking for people who'd be able to work full-time (remotely), with compensation of $70/hour. I expect each project to take 4-8 months to complete and lead to a first-author publication at a top-tier ML or NLP conference.

Below are a few examples of projects I have in mind:

1.  Pretraining language models on more factual, high-quality, and value-aligned text, so that they are [easier to control](https://arxiv.org/abs/2105.11447) and [more truthful](https://arxiv.org/abs/2109.07958).
2.  Developing algorithms for training language models to learn from different forms of human feedback, beyond simple scalar/numerical feedback as in [prior work](https://arxiv.org/abs/2009.01325).
3.  Finding tasks that exhibit *inverse scaling,* where better performance on the language modeling pretraining objective leads to worse downstream task performance. Examples of such tasks include [TruthfulQA](https://arxiv.org/abs/2109.07958) and [RealToxicityPrompts](https://arxiv.org/abs/2009.11462).

If what I've described sounds like a good fit, I'd love to hear from you over email! Just send me your website, resume, LinkedIn, GitHub, Google Scholar, or anything else that might be helpful for understanding your background (my email is perez at nyu dot edu).
# Mystery Hunt 2022

[Teacher's password](https://www.lesswrong.com/posts/StbTTmFZD6jZrCCH8/teacher-s-password-the-lesswrong-mystery-hunt-team) will be participating in the [2022 MIT Mystery Hunt](https://www.mitmh2022.com)!

**What is the MIT Mystery Hunt?**

The MIT Mystery Hunt is a puzzle-solving competition, in which teams race to solve a series of puzzles without instructions. Here is a quote by Mike Develin about puzzle hunts in general:

> Imagine a word search.
> 
> Now imagine you aren’t told what words to look for.
> 
> Now imagine you aren’t told it’s a word search.
> 
> Now imagine it isn’t a word search.

You can learn more about puzzle hunts [here](https://blog.vero.site/post/puzzlehunts), or the MIT Mystery Hunt [here](https://www.mit.edu/~puzzle/).

**When is the MIT Mystery Hunt?**

The Mystery Hunt is from 9AM PT on Friday, January 14, 2022 to 9AM PT on Monday, January 17, 2022 (MLK Jr. Day weekend).  Depending on how many people sign up, (and where they are) we might be solving puzzles for that entire 72 hours. You can log on and work on puzzles however much you like.

**Where is the MIT Mystery Hunt?**

Usually, the mystery hunt is on MIT campus, but this year, it is entirely remote. We will be using three main tools to collaborate: Discord, Gather Town, and Google Docs. It seems likely that there should be a Berkeley in-person group. Let me know if you want to take responsibility for that.

**What is Teacher's Password?**

Teacher's Password is the name of LessWrong's Mystery Hunt team. The team is open to anyone on LessWrong, and friends of anyone on LessWrong, including people who are new to puzzle hunts. Our team emoji is 📎. We are not going to win.

**How do I know if I will like the Mystery Hunt?**

We will be casually doing the starrats mini practice hunt starting 9AM PT on Tuesday, December 14, 2021. (This is the official prologue to the 2022 MIT Mystery Hunt.) I don't know how long the mini hunt will take, but I suspect we will still be working on it for more than a day.

**How do I Sign Up?**

Join the [discord](https://discord.gg/4KAVx6bTks). (This is a different discord from last year.) Just joining the discord is sufficient for signing up, and I will remove the discord link before the hunt starts.
# ARC is hiring!

The [Alignment Research Center](https://alignmentresearchcenter.org/) is hiring researchers; if you are interested, please apply!

(**Update: We have wrapped up our hiring "round" for early 2022, but are still accepting researcher applications on a rolling basis** [**here**](https://docs.google.com/forms/d/e/1FAIpQLSegoNiBwfhZN3v0VkBGxKx6eYybSyWo-4WFHbkMnyXaMcIZeQ/viewform)**, though it may take us longer to get back to you.**)

### What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read [Eliciting Latent Knowledge](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit?usp=sharing), a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

### Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples [described in our report on ELK](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit?usp=sharing).

We’re **open to anyone who is excited about working on alignment even if you don't yet have any research background** (or your research is in another field). You may be an especially good fit if you:

*   Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).
*   Have experience designing algorithms, proving theorems, or formalizing concepts.
*   Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).
*   Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

### Hiring process and details

You can apply by filling out [this short form](https://forms.gle/4upi1CqVxipuPbRK9). We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

We are based in Berkeley, CA and would prefer people who can work from our office, but we’re open to discussing remote arrangements for great candidates.

Salaries are in the $150k-400k range depending on experience.
# ARC's first technical report: Eliciting Latent Knowledge

ARC has published a report on [Eliciting Latent Knowledge](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit?usp=sharing), an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.

The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to [ontology identification](https://arbital.greaterwrong.com/p/ontology_identification/) (and [other](https://www.alignmentforum.org/posts/gQY6LrTWJNkTv8YJR/the-pointers-problem-human-values-are-a-function-of-humans) [similar](https://intelligence.org/files/AlignmentMachineLearning.pdf) [statements](https://www.alignmentforum.org/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1)). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.

The report is available [here](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit?usp=sharing) as a google document. If you're excited about this research, [we're hiring](https://www.alignmentforum.org/posts/dLoK6KGcHAoudtwdo/arc-is-hiring)!

### Q&A

We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.

*Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.*
# Robin Hanson's "Humans are Early"

This post was nominated for the 2021 Review by Algon. The post is short, so I'm including the full text:

> Imagine that advanced life like us is terribly rare in the universe. So damn rare that if we had not shown up, then our region of the universe would almost surely have forever remained dead, for eons and eons. In this case, we should still be able to predict *when* we humans showed up, which happens to be now at 13.8 billion years after the universe began. Because we showed up on a planet near a star, and we know the rate at which our universe has and will make stars, how long those stars will last, and which stars where lived far enough away from frequent sterilizing explosions to have at least a chance at birthing advanced life.
> 
> However, this chart (taken from our [new paper](https://arxiv.org/abs/2102.01522)) calculates the percentile rank of our current date within this larger distribution. And it finds that we are surprisingly early, *unless* you assume *both* that there are very few hard steps in the evolution of advanced life (the “power n”), *and also* that the cutoff in lifetime above which planets simply cannot birth advanced life is very low. While most stars have *much* longer lives, *none* of those have any chance whatsoever to birth advanced life. (The x-axis shown extends from Earth’s lifetime up to the max known star lifetime.)
> 
> [![](https://www.overcomingbias.com/wp-content/uploads/2021/02/earlyness_k0p12d4.png)](http://www.overcomingbias.com/wp-content/uploads/2021/02/earlyness_k0p12d4.png)
> 
> In the paper (in figures 2,17), we also show how this percentile varies with three other parameters: the timescale on which star formation decays, the peak date for habitable star formation, and a “mass favoring power” which says bu how much more are larger mass stars favored in habitability. We find that these parameters mostly make only modest differences; the key puzzle of humans earliness remains.
> 
> Yes, whether a planet gives rise to advanced life might depend on a great many other parameters not included in our calculations. But as we are only trying to estimate the date of arrival, not many other details, we only need to include factors that correlate greatly with arrival date.
> 
> Why have others not reported the puzzle previously? Because they neglected to include the key hard-steps power law effect in how chances vary with time. This effect is not at all controversial, though it often seems counter-intuitive to those who have not worked through its derivation (and who are unwilling to accept a well-established literature they have not worked out for themselves).
> 
> This key fact that humans look early is one that seems best explained by a grabby aliens model. If grabby aliens come and take all the volume, that sets a deadline for when we could arrive, if we were to have a chance of becoming grabby. We are *not* early relative to that deadline.
# Leverage

If you want to change the world then you need to act on scale. A single human being cannot act on scale directly. You need leverage. **Leverage** is mechanism by which you can cause an impact several orders of magnitude bigger than yourself.

Sam Altman [gives five examples of leverage](https://blog.samaltman.com/how-to-be-successful): delegation, capital, brand, network and technology.

# Delegation

The most obvious way to obtain leverage is to manage other people. A disadvantage of managing people is cost. Salaries are expensive. Communication is expensive. Your employees interests are often unaligned with your own.

The biggest disadvantage of managing people is that some tasks cannot be parallelized. A hundred mediocre singers cannot work together to write [*You Belong With Me*](https://www.youtube.com/watch?v=VuNIsY6JdUw) or [*Thunder Road*](https://youtu.be/UDIDawmeeI0). A hundred mediocre writers working together could not produce *Harry Potter* or *The Fault in Our Stars*. A hundred mediocre *Age of Empires II* players could not work together to compete with TheViper because only one player is allowed at the keyboard at the time.

# Capital

If you want Carly Rae Jepsen to produce a new hit song then the first thing you should do is throw as much money at her equipment as you can. Carly Rae Jepsen shouldn't just have the best microphone in the world. She should have an entire sound crew setting things up for her. She should have a limo taking her wherever she needs to go. Carly Rae Jepsen is a bottleneck. Any time she spends thinking about sound or makeup or scheduling is time she's not spending writing or singing or practicing songs.

Buying infrastructure is optimal when you have plenty of money and are bottlenecked by talent.

# Brand

Brand is when people trust a symbol. Trust reduces transaction costs. The disadvantage of brand is it cannot be bought; trust must be earned.

# Network

Networks are another form of trust. Usually the hardest part of building a network is bootstrapping it. After that, you can grow your network organically with little effort.

# Inventing Technology

Buying technology is amassing capital. Inventing technology creates a way to amass capital. Technological invention is leverage squared.

The disadvantage of technological invention is it is hard to do and hard to delegate. But if you can do it, technological invention is among the highest leverage activities there is.
# More power to you

When people say things like “we were promised [flying cars](https://rootsofprogress.org/where-is-my-flying-car),” I sometimes wonder, “who *promised* this?”

I guess this is what they mean. From a 1959 ad that ran in the *LA Times:*

![As a friend pointed out, “they're not even wearing seat belts!”](https://rootsofprogress.org/img/flying-car-ad.jpg)

*As a friend pointed out, “they're not even wearing seat belts!”* [*The Los Angeles Times, June 21, 1959*](https://www.newspapers.com/clip/90628174/flying-car-ad/)

“They’re working on it!” the ad claims. “Some of this is happening already.” (Implying, of course, that some of it was still pretty speculative.)

The ability to “dial” a book, lecture, or demonstration is here; the ultrasound dishwasher, automatic bed-maker, and flying car sadly are not.

But here’s what’s most interesting to me: First, the reference, without explanation or justification, to “tomorrow’s higher standard of living”—something people simply assumed was coming. Second, that it was uncontroversial that this higher standard of living would require more electricity. Finally, the boasting of doubling electricity production in ten years. As I’ve pointed out, society once had a different fundamental [attitude towards growth](https://rootsofprogress.org/indignation-at-the-1890-census).

The US did, in fact, approximately double electricity generation every decade from 1950 until 1973—the year of the oil crisis—after which electricity growth never really recovered. It took almost thirty years to double again, from 1973 to 2000, during which time it was just keeping up with population growth; that is, per-capita energy usage was not increasing. And since 2005, even the total amount produced has been flat:

![US electricity generation](https://rootsofprogress.org/img/us-electricity-generation.svg)

*US electricity generation.* [*Data from US Energy Information Administration*](https://www.eia.gov/energyexplained/electricity/electricity-in-the-us-generation-capacity-and-sales.php)

“The electric companies are resolved,” the ad says, to maintain for America “the best electric power service in the world.” Well, we are still [one of the top overall producers, second only to China](https://ourworldindata.org/grapher/electricity-generation?country=IND~USA~RUS~CHN~JPN):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c53c4eb892fd70be9d4b95347f347c55eb06dd78556a9366.png)

But since the oil shocks, I’d say that “resolve” has been shaken.