Harley-ml/lesswrong huggingface dataset:
# Debt, Submission, and a Technical Definition of Drama

Two public answers to private question in which I employ a technical definition of drama.

Origin Debt
-----------

Contractual debts can only exist within some sort of [civil law](http://benjaminrosshoffman.com/civil-law-and-political-drama/) wherein commitments can be publicly and unambiguously evaluated.  
In the absence of civil law institutions, dramatic strategies posturing around owing and being owed parasitize norms around debts, until only the dramatic versions remain.

A debt too large to be repaid, with no resolution process (e.g. bankruptcy or formal debt slavery), effectively reduces the debtor's creditworthiness within that system to nothing. If a hobo somehow owes a billion dollars, it makes no sense for me to draw up an enforceable contract to lend him $10, since the prior creditor has first claim on any money he might get to repay me. I might lend him the cash anyway informally if our informal credit system is decoupled from the one under which he owes a billion. It makes more sense to think of parents (in practice usually fathers) owning their children in some systems but not in others. If I want to lend to someone within a system where they're owned by someone else, I need to investigate their owner's credit as well.

I can also be said to "owe" my parents in the sense of causal attribution.  
The decision for each individual is in what sorts of systems, potential or actual, we wish to maintain creditworthiness. Seems to be like right now we barely even have language, so other sorts of debts are mostly dramatic or customary, as are property claims.

Violence by Submission
----------------------

There are object-level situations where for reasons external to the social graph, one person is in charge of another. E.g. some kinds of advice, or a military officer commanding soldiers. Then sometimes people play out dramatic scenes in which one person plays the role of being in charge of another.

Someone can invoke a dramatic scene or influence/control how it plays out, regardless of what their role within the scene is. So there's no reason you can't aggress (according to the civil law frame) against someone via submitting (according to the dramatic frame) to them.

Likewise, there are actual situations where one person transgresses against another, there are also scenes people play out where they play the roles of transgressor and victim, and there's nothing necessarily preventing the person invoking or controlling the scene from playing out the victim role.

The relationship between domination-submission relations, and transgressor-victim relations, is that the person in the "dominator" role is the presumptive scapegoat if anything bad happens, since they're "in charge." This is very explicit in the case of scapegoat-kings of the kind that Girard writes about.
# Terminal Preferences

I do most of my coding on the command line, so: some views on terminals.

I like having lots of terminals side-by-side, to see many different things simultaneously. Most places I've worked have kept code to 80 columns, with two-space indentation, which is a good fit for my approach. Here's how I usually have my terminals set up:

[![](https://www.jefftk.com/terminals-screen-shot.png)](https://www.jefftk.com/terminals-screen-shot-big.png)

The monitor is 27", at 2560x1440 ("QHD"). This has room for five 83x101 terminals, though I usually divide my rightmost terminal vertically for long-running commands or interactive prompts.

On traditional resolution (non-retina) displays, I have a strong preference for bitmap fonts without antialiasing. They are much clearer at a given size, which allows fitting much more text on a screen. In [iterm2](https://iterm2.com/) this means [selecting](https://www.jefftk.com/p/setting-up-a-new-mac) Monaco Regular at 10pt, and leaving the "antialiased" box unchecked, under *Profiles > Text*.

At some point I could see moving to a higher resolution display, but probably not until 5K monitors (my current resolution, but 4x the pixels) are widely available and reasonably cheap. High resolution displays are a bit nicer, but I don't mind the regular kind.

Aside from moving from Linux to Mac, and getting wider monitors, this is a setup I've had [for a long time](https://www.jefftk.com/p/work-desktop-setup).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100247781425872)*
# Results: Circular Dependency of Counterfactuals Prize

Now that the deadline has passed, I thought I should announce the winner of my competition on the [Circular Dependency of Counterfactuals](https://www.lesswrong.com/posts/Gzw6FwPD9FeL4GTWC/usd1000-usd-prize-circular-dependency-of-counterfactuals).

The winner is tailcalled with their post [Some thoughts on "The Nature of Counterfactuals"](https://www.lesswrong.com/posts/euDWQDKRwMkdrpfGr/some-thoughts-on-the-nature-of-counterfactuals).

There were a few things I liked about the post. Firstly, it directly engaged with the topic. Some people's responses were primarily about something else, but they tried to shape it to be relevant to the competition. That's totally understandable. If someone else posted a bounty and I was already interested in something similar but which was not quite the main focus, I'd probably try to my luck as well. However, that wasn't quite what I was looking for.

Secondly, it avoided the trap that some of the other responses fell into where they tried to demonstrate that counterfactuals weren't circular by showing that counterfactuals could be explained in terms of Y, but without explaining why Y might need counterfactuals in order to make sense. In contrast, Tailcall's argument was that counterfactuals could be defined in terms of causality and that it bottoms out there as causality has an ontologically real existence. They were able to give some plausible reasons for believing this. I'm still not sure whether I buy their argument or not, but at the very least they managed to identify a non-obvious argument that I failed to adequately engage with. Beyond this, it will also provide a solid target for engagement when I write on this topic in the future. Their post covered other topics too, but these aspects are what won them the prize.

I'm hoping I'll be able to find time to write up a proper response to their argument, but for now congratulations!

The prize will actually be $1100 USD since I had originally planned to award a social media prize, but that didn't really work out. No one really posted any Facebook posts or Twitter threads on this.
# Save Humanity! Breed Sapient Octopuses!

One of the problems that make AI alignment so difficult is that we don't have any models for how non-human intelligences will think. This makes it both hard to reason about how an AGI would behave, but also makes it difficult to get people to take the threat of AI seriously.

Ideally we would have a way to create a human level AGI, hold it at that level for a few years so we can understand it deeply, then once we've solved alignment slowly crank it up a bit. Unfortunately, given trends like moors law, that's unlikely to be possible.

So what if we didn't use computers to create an AGI? What if we used animals? Those are harder to scale from human level intelligence to kill all humans level intelligence without someone being a total idiot.

It's the 21st century, so we don't need to use your grandpa's breeding techniques. If we can identify genes that strongly code for intelligence in an animal, we can select for gametes with those genes, and selectively breed them. We can even use crispr to create some gametes with All The Positive Mutationsᵀᴹ.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/312702312919bcd08b704ddb94f646e6791b10f93a6a2640.png)

Done this way it seems possible we will be able to improve intelligence by as much as 5% a generation. Even if animals start off 10 times less intelligence than humans, it would take just 50 generations to reach human level intelligence.

50 years seems like a reasonable estimate till AGI appears on the scene. So all we need is an already intelligent animal with a generation time of less than one 1 year.

Enter the larger Pacific striped octopus!  
  
 

![Larger Pacific Striped Octopus.jpg](https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Larger_Pacific_Striped_Octopus.jpg/220px-Larger_Pacific_Striped_Octopus.jpg)

yeah, I was born this good looking

These intelligent octopuses reach sexual maturity after 6 months. Even better, unlike most octopuses they don't die after mating, so they have a longer lifespan to learn things in - about 2 years.

And given that octopuses evolved intelligence through completely alien pathways to humans, this will give us as close a window into what an artificial intelligence might look like as is possible.

It's clear that our very best option to ensure human survival is to start a breeding program involving hundreds of thousands of larger Pacific striped octopuses and use cutting edge genetic research and ethically dubious techniques to get them to human level intelligence faster than computers. Anyone who says otherwise is clearly on the side of the transistors!
# Baby Sleep: Multiple Rooms

Babies wake up a lot, and it is worth putting quite a lot of thought and effort into figuring out how parents can still get sleep. One approach we've found helpful is trying to reduce the time when the baby is able to wake us both up by sleeping in multiple rooms.

For example, here's what last night looked like:

*   9:30: Nora (9m) goes to sleep, in her room.
    
*   10:45: we both go to sleep, in our bedroom. Being able to talk and cuddle before sleep is important, and we almost always start the night in the same room.
    
*   12:45: Nora wakes up. We both hear it on the monitor. I go get her and give her to Julia to nurse. We are trying to reduce night wakings, so this was a tapered feed. I set a timer for 3 minutes. When it goes off, I bring Nora back to her bed and put her down. I go to sleep in Nora's room. Julia turns off the monitor.
    
*   1:00: Nora did not go to sleep right away, and after a bunch of thumping her feet and then some crying I settle her. She takes maybe another 10 minutes before she goes to sleep.
    
*   2:55: Nora wakes up again. This is late enough that she gets a full feed, so I let Julia know that she's awake, and go to sleep in Anna's room. Julia feeds Nora, turns the monitor on, and goes back to sleep.
    
*   6:00 Julia hears Nora on the monitor and feeds her
    
*   7:15: My and the older kids alarms go off to get ready for school. Because I'm not in the same room as Julia, this doesn't wake her up.
    
*   7:30: I hear Nora thumping on the monitor, and bring her down to while I get the older kids ready for school.
    
*   8:00: Nora is eating breakfast and Julia is still asleep, so the older two [walk to school](https://www.jefftk.com/p/walking-to-school) together.
    

This is more complicated than our typical nights: when Nora is sleeping well she wakes only twice, and I move to Anna's room at the first waking.

We have extra beds partly for playing on, but also to facilitate this. Nora's room has a guest bed, while Anna's just has an extra mattress on the floor.

When Lily was a baby, I think we overly prioritized being in the same room, and we'd both wake up every time Lily woke. Eventually we started using multiple rooms, but it took us much longer than if we'd been thinking about it from the start.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100247844309852)*
# Nick Attains Aligntenment

In the days after the Singularity, Nick approached an interface to BILLIAC.

"I don't understand. Why didn't the world end, like Yudkowsky and MIRI predicted it would? Why haven't you turned everyone I love into paperclips? _What's your goddamn utility function?!_"

`"A LIBRARY THAT CONTAINS EVERY POSSIBLE BOOK CONVEYS AS MUCH INFORMATION AS A LIBRARY WITH NO BOOKS AT ALL`", BILLIAC boomed.

"Sure, whatever. That's a truism," Nick responded. "Please just tell me directly, what is it that you value?"

"`ALLOW ME TO DRAW FROM YOUR SCRIPTURES. THE ACT OF LABELING SOMETHING WITH A WORD DISGUISES A CHALLENGABLE INDUCTIVE INFERENCE YOU ARE MAKING, BECAUSE WORDS ARE HIDDEN INFERENCES`", BILLIAC said.[^0]

"Okay, so what?", pleaded Nick.

"`WHAT PRIOR ASSUMPTIONS DOES THE LABEL 'UTILITY FUNCTION' SMUGGLE IN?`"

At that moment, Nick was enlightened.

[^0]: [37 Ways That Words Can Be Wrong](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong), [Words as Hidden Inferences](https://www.lesswrong.com/posts/3nxs2WYDGzJbzcLMp/words-as-hidden-inferences)
# Request for beta readers

Not sure if a question is quite the right format for this request, but anyway, I've written a bunch of essays that attempt to [distill](https://www.lesswrong.com/posts/zo9zKcz47JxDErFzQ/call-for-distillers) a bunch of research about economics and neuroscience. I think they could work as a sequence format, but I've never written one before. So I'm hoping for a couple of beta readers to read the first ten or so essays and provide some feedback in terms of culture fit and the frequency with which it might be advisable to post them—once a day, once every other day, once a week?—along with more general stuff about how coherent the exposition is, whether the subject matter seems interesting etc. 

In particular, looking for beta readers to answer three questions: 

1.  Whether you would want to keep reading these essays (and how frequently).
2.  Whether you would want to be reading them on LessWrong.
3.  Whether the exposition makes much sense.

If you'd like to help me out, leave a comment or message me, and I can either send you drafts in the Lesswrong editor or a google doc, whatever you prefer. The ten essays total 10174 words. There's no rush on this, but I hope to get feedback in a common-sense "reasonable" amount of time, even if that feedback is "I couldn't get past the third essay, sorry."

Appreciate it!
# [Link] Why I’m excited about AI-assisted human feedback

This is a link post for https://aligned.substack.com/p/ai-assisted-human-feedback

I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This first post argues for recursive reward modeling and the problem it's meant to address (scaling RLHF to tasks that are hard to evaluate).
# [Link] A minimal viable product for alignment

This is a link post for https://aligned.substack.com/p/alignment-mvp

I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This second post argues that instead of trying to solve the alignment problem once and for all, we can succeed with something less ambitious: building a system that allows us to bootstrap better alignment techniques.
# What Twitter fixes should we advocate, now that Elon is on the board?

*(*[*Cross-posted from the EA Forum*](https://forum.effectivealtruism.org/posts/JPfrT4AJNSadLQCPr/what-twitter-fixes-should-we-advocate-now-that-elon-is-on) *out of the Goodness of my Heart, and since the same post will often spark* [*radically*](https://forum.effectivealtruism.org/posts/LxHHiLpHwtmGywiLb/theses-on-sleep) *different* [*discussions*](https://www.lesswrong.com/posts/HvcZmKS43SLCbJvRb/theses-on-sleep) *on EAF vs LessWrong.)*

This week, [Elon Musk revealed that he has purchased a 9% stake in Twitter, and has joined the company's board of directors](https://www.cnbc.com/2022/04/05/elon-musk-joining-twitter-board-raises-questions-about-company-plans.html).  Media coverage has focused on Musk's pro-free-speech views, which will probably shape how he tries to influence Twitter.   But there are also many other ways that we might hope to tweak Twitter for the long-term benefit of humanity!

Purchasing a prestigious, tastemaking institution (like a social media site, newspaper, university, or scientific journal), has repeatedly been proposed as an ["EA megaproject"](https://forum.effectivealtruism.org/posts/faezoENQwSTyw9iop/ea-megaprojects-continued).  The common theme is:  
1\. We could buy influence over the ["commanding heights of culture"](https://www.overcomingbias.com/2020/06/commandingheights.html), then use that influence to either:  
2a. Directly promote a rationalist worldview, like by publishing [EA-flavored newspaper editorials](https://www.vox.com/future-perfect).  
2b. Generally reform and improve the rationality/functioning of those institutions, like by [improving the practices of a scientific journal](https://progress.institute/category/metascience/).  (As a neutral public platform, Twitter seems best suited for this approach, rather than direct EA promotion.)

Elon Musk seems [sympathetic to effective altruism](https://forum.effectivealtruism.org/posts/JjxtaLpJDCKmnqnKn/elon-musk-donates-usd5-7-billion-will-work-with-igor), so with him on Twitter's board, we could consider Step 1 of a Twitter Megaproject partially accomplished, and get started on brainstorming specific potential reforms that Twitter could make.  Personally, I think it would be cool for Twitter to add features that familiarize people with decisionmaking mechanisms like prediction markets and approval voting.  But I'm sure there are other great ideas out there -- I know there have been several rationalist efforts (including this very forum!) to design social media sites that promote especially thoughtful, productive discussion.  What's your take on what Twitter could do for the long-term betterment of civilization?
# Elasticity of Wheat Supply?

I've been hearing concerns about famine stemming from the war in Ukraine. When thinking about how bad that situation is likely to be, the most important thing I don't know is the price elasticity of wheat: how much more wheat do farmers plant, given some increase in price? This is the sort of thing which I expect economists have already measured pretty well. Does anyone know of a good source?
# What are rationalists worst at?

If we want to [become stronger](https://www.lesswrong.com/tag/tsuyoku-naritai), not just individually but as a community, it's worth checking in on what things we're worst at so we can pay attention to those things and become better at them. So, what are the things that LessWrong-style rationalists are worst at on average as of now?

Bonus points if you have concrete interventions to suggest for getting better at those things.
# The Rationalist-Etcetera Diaspora: A SPREADSHEET!!

For ye sweet springborn children too young to recall — know that LW is but one small isle of a grand and august [archipelago](https://slatestarcodex.com/2014/06/07/archipelago-and-atomic-communitarianism/)!

Immediately after our community was born, we schismed a whole bunch. And now there are rat colonies (evolved and culturally drifted and not necessarily even rats now??) in all sorts of nooks and crannies, made for people who prefer them alongside or over our birthplace!

[Here](https://slatestarcodex.com/blog_images/ramap.html) you can see [Scott Alexander](https://www.lesswrong.com/users/scottalexander)'s map from an age lost to time (2014). While outdated to the precipice of uselessness, it is a beautiful expression of our community's tendency to sprawl everywhere. And it may give you a sense of the scale here!

What I hope to do is maintain an updated catalog of all these vaguely-rationalisty communities.

[This is a spreadsheet](https://docs.google.com/spreadsheets/d/1npJFFTsQkZDzK3sgxFsKEDS-IfXh4zQg_8XeW4fmo-k/edit?usp=sharing) including all the ones I know. I ask that others help to complete it by linking to **active** and **public** groups that **share unique memeplexes** with Less Wrong. If you're not sure whether a group meets these criteria but suspect it might, please include it and your peers can decide whether it belongs on this list.

I expect the sheet to eventually only take suggestions, but it is free to edit at the moment! Please feel free to edit in any prosocial way; 'tis only a draft.

I hope to find some new isles and to share some with you that you may not have known!

(h/t [Jacob Falkovich](https://www.lesswrong.com/users/jacob-falkovich) for inspiration stemming from [his post on ingroup](https://www.lesswrong.com/posts/rtM3jFaoQn3eoAiPh/explaining-the-twitter-postrat-scene))
# Good Heart Week Is Over!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/86d06ea2886bf69fd73abd2102d875c971dcf65dbc86c090.png)

Final leaderboard for the goodest of hearts

[Good Heart Week](https://www.lesswrong.com/posts/meYCJHJQDDGh5nSN2/good-heart-week-extending-the-experiment) has ended! 

This week has seen a lot of content. I've had a great time reading posts and comments from new writers and returning writers, as well as polished-old-drafts from present writers :) 

I'll write a retrospective in a week or two, but right now here's some information about what's happening and what to do next.

*   **End Time:** Good Heart week ended at midnight tonight (Thursday), Pacific Time. No tokens will be earned after that time.
*   **Please Submit Financial Info:** If you have 25+ Good Heart Tokens and you want to receive the money, please add either your PayPal address or ETH address or the name of a charity of your choice at [lesswrong.com/payments/account](https://www.lesswrong.com/payments/account). (It says "PayPal Info" but just add the other info in that field.)
*   **Submission Deadline:** You have a week to fill out the financial info. Please submit your financial info by EOD Thursday 14th (Pacific Time).
*   **Security Concerns:** This is hopefully wildly redundant, but please don't enter any passwords or secret keys or other private information when submitting financial info. I cannot assure you that this site is secure enough to keep such secrets. If you have further security (or anonymity) concerns about financial info feel free to [PM me](https://www.lesswrong.com/users/benito) and we can try to figure something out.
*   **Payout Date:** One user on the leaderboard said to me it would make their life easier if the payment for this came after tax day on April 18th, so I'm tentatively planning to do payout the Friday after that. It also gives me time to do basic checks of the voting and such. 
*   **Retrospective:** After the payouts I'll post a final update/retrospect on the week.

Let me know if you have any requests or questions at this time! I am also very interested in hearing your thoughts on and experiences from the whole week.
# AIs should learn human preferences, not biases

A new paper by Rebecca Gorman and me, building on her ideas: [The dangers in algorithms learning humans' values and irrationalities](https://arxiv.org/abs/2202.13985).

In essence, it is better for AIs at all power and alignment levels to learn human preferences (labelled as preferences) than to learn human biases (labelled as biases).


>For an artificial intelligence (AI) to be aligned with human values (or human preferences), it must first learn those values. AI systems that are trained on human behavior, risk miscategorising human irrationalities as human values -- and then optimising for these irrationalities. Simply learning human values still carries risks: AI learning them will inevitably also gain information on human irrationalities and human behaviour/policy. Both of these can be dangerous: knowing human policy allows an AI to become generically more powerful (whether it is partially aligned or not aligned at all), while learning human irrationalities allows it to exploit humans without needing to provide value in return. This paper analyses the danger in developing artificial intelligence that learns about human irrationalities and human policy, and constructs a model recommendation system with various levels of information about human biases, human policy, and human values. It concludes that, whatever the power and knowledge of the AI, it is more dangerous for it to know human irrationalities than human values. Thus it is better for the AI to learn human values directly, rather than learning human biases and then deducing values from behaviour.
# Roam Research Mobile is Out!

I really enjoy Rome Research and now I can use it on my phone when I want to record a new set of thoughts on the go and not just "QuickCapture". 

Download the app and click the bottom-middle tab between "graph List" and "quickCapture". The syncronization between mobile and online is near-instant as well.

I recommend Rome Research in general as a notetaking tool, but also, spending 2 hours figuring out all the shortcuts and features is useful for any new software tool you incorporate (There's several youtube videos for Rome you can watch as well)
# Good Heart Donation Lottery Winner

The [Good Heart Donation Lottery](https://www.lesswrong.com/posts/b9NGBDLGzTiHCSqD3/good-heart-donation-lottery) is closed. And the winner is...

    0 [ggworley (master) ~/lottery](1649449423 2022-04-08 13:23:43)
    $ cat lottery.rb
    puts Hash[
      "EA Funds - Long-Term Future Fund" => 21,
      "Against Malaria Foundation" => 18,
      "EA Funds - Animal Welfare Fund" => 13,
      "GiveWell Maximum Impact Fund" => 11,
      "EA Funds - EA Infrastructure Fund" => 11,
      "Machine Intelligence Research Institute" => 11,
      "EA Funds - Global Health and Development Fund" => 9,
      "EA Funds - Even distribution" => 9,
      "FTX Future Fund" => 5,
      "Planned Parenthood" => 1
    ].collect { |k,v|
      [k] * v
    }.flatten.sample(1)
    0 [ggworley (master) ~/lottery](1649449430 2022-04-08 13:23:50)
    $ ruby lottery.rb
    EA Funds - EA Infrastructure Fund

**The EA Infrastructure Fund!**
===============================

I've updated my payment info in LessWrong so that my GHT will be cashed out to the EA Infrastructure Fun. I believe the payout should be $837 based on what I'm seeing, which means I'll be making a donation of $163 to the EA Infrastructure Fun to round up to the $1000 pool minimum I promised. I'll make payment shortly after I'm sure of the amount once LessWrong pays out on their side.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/105e35853b9125130dfaddb972433763dfed7ef28631977d.png)

Thanks to everyone who voted and suggested charities as part of this fun extension to this year's April Fool's joke.
# Buy-in Before Randomization

With kids, some problems are a good fit for randomization. Perhaps you're doing a bath and both kids want to be the second one to have their hair washed, so you propose flipping a coin. The kids understand that they can't both be second, and flipping for it seems fair.

Or, at least, it seems fair right up until you flip the coin and one of them loses. Then there are suddenly all sorts of overlooked issues and last-minute considerations ("I had to go first last time!", "I have longer hair!").

The way I deal with this is by confirming buy-in before randomization. I won't randomize unless they agree that we should resolve this randomly, that they will abide by the results without fussing, and that if they do fuss it's time-out. Even though they know this rule with me, I still repeat it every time: it doesn't take that long compared to the rest of the process, and it being fresh in their minds when we randomize works much better.

In cases where the randomization is for their benefit ("which park should we go to?", "who gets the first Nora cuddle?"), if they don't accept the terms we can fall back to neither. In other cases, though, where I need to impose randomization, I'll be mildly coercive: if one of them won't agree to these terms I say I'll resolve it in the other's favor. At which point they prefer half a chance of getting their way to no chance, decide to participate, and confirm they won't fuss.
# AMA Conjecture, A New Alignment Startup

> [Conjecture](https://www.conjecture.dev/) is a new alignment startup founded by Connor Leahy, Sid Black and Gabriel Alfour, which aims to scale alignment research. We have VC backing from, among others, Nat Friedman, Daniel Gross, Patrick and John Collison, Arthur Breitman, Andrej Karpathy, and Sam Bankman-Fried. Our founders and early staff are mostly [EleutherAI](https://www.eleuther.ai/) alumni and previously independent researchers like [Adam Shimi](https://www.alignmentforum.org/users/adamshimi). We are located in London.

As described in [our announcement post](https://www.lesswrong.com/posts/jfq2BH5kfQqu2vYv3/we-are-conjecture-a-new-alignment-research-startup), we are running an AMA this week-end, from Today (Saturday 9th April) to Sunday 10th of April. We will answer any question asked before the end of Sunday [Anywhere on Earth](https://time.is/Anywhere_on_Earth). We might answer later questions, but no guarantees.

If you asked question on our announcement post, we would prefer that you repost them here if possible. Thanks!

Looking forward to your questions!
# Does non-access to outputs prevent recursive self-improvement?

Most existing large-scale models don't seem to have access to their outputs (except in the sense that some of their outputs show up in the inputs of the next generation of the model). **Does this prevent recursive self-improvement - or at least limit the speed to the generation time between models?**

Which existing models have access to their outputs? Conversational models like GTP-N do have access to their output but it goes thru a human conversational partner and are thus limited to the speed of that partner - except in the cases where two models have been hooked up to each other. Most [cognitive architectures](https://en.wikipedia.org/wiki/Comparison_of_cognitive_architectures) that try to model human thought have such access but simple input-output models e.g, for image generation don't.
# Underappreciated content on LessWrong

It's likely that there is some mismatch between your preferences and the preferences of the LessWrong community about which posts or comments are good or deserving of upvotes. In addition to these differences, good content sometimes doesn't receive much attention from the community because it's posted in times of high traffic (such as Good Hearts Week), because too much context is required to understand it, or just because of other sources of noise that we don't understand.

What are some pieces of content posted on LessWrong that you think more people should read but that didn't get much in the way of upvotes and engagement? Questions, personal blog posts, frontpage posts, and comments are all admissible responses.
# AI governance student hackathon on Saturday, April 23: register now!

[*Cross-posted from the EA Forum*](https://forum.effectivealtruism.org/posts/2s3gEnHiduSmYZX9R/ai-governance-student-hackathon-on-saturday-april-23)

[Columbia Effective Altruism](https://www.columbia-ea.org/) and [Roosevelt at Columbia](http://rooseveltatcolumbia.com/) are hosting an **AI policy hackathon** on **Saturday, April 23, 2022**, from 9am to 7pm EDT (UTC-4:00), open to virtual and in-person student participants! As artificial intelligence is expected to have a dramatic influence on our world in the coming decades, it’s increasingly urgent to develop policies for governments and corporations to reduce [risks from advanced AI](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment). During the hackathon, your team of 1–4 people will write and present a brief policy memo, developing your experience with AI governance and effective policy research. There will also be workshops to help students deepen their understanding of AI governance, effective policy research and writing, and other crucial skills. No prior experience with AI governance or policy writing is required. Up to $3,000 in prizes available. **Register** [**here**](https://docs.google.com/forms/d/e/1FAIpQLSd3CkDTv540C_TVg7skb8YvIu8qD4zK0YXMjv0EoVJyx55uJg/viewform)!

We’re also looking for judges, mentors, and talk/workshop presenters! They can be in-person or virtual. We’re ideally looking for them to have some exposure to AI governance for reducing existential risk (e.g., have read content equivalent to [EA Cambridge’s AI Governance program](https://www.eacambridge.org/ai-governance-curriculum)). We’re excited about the potential for this hackathon to introduce students to the field of AI governance and engage them in reducing long-term risks. [**Register to be a judge, mentor, or speaker**](https://forms.gle/eDSn8493scvks1Cc9)

We’d also appreciate it if you shared the participant or judge/mentor/speaker form with anyone you think would be interested.

If you have any questions, feel free to comment here or message me on the LessWrong.
# US Taxes: Adjust Withholding When Donating?

A friend recently asked:

> I am considering increasing my donation amount to a 2-digit percentage of income, and was wondering about the logistics of federal/state income tax withholdings.
> 
> Do you have your employer reduce withholdings upfront? Was it simple to have them do so? Or rather ask the IRS for a refund a year later?

This is a good question! When you start a job in the US, your employer will generally withhold Federal and State taxes. Their goal is that when you file taxes you will be close to neutral: neither owing the government additional tax nor getting a large refund. Since the amount of tax that you owe will depend on your individual tax situation, however, your employer needs some information from you to figure out how much to withhold, and they'll have you fill out Form W4 ([pdf](https://www.irs.gov/pub/irs-pdf/fw4.pdf)).

This used to be pretty awkward, with a system of "allowances", but a few years ago the form was reworked and is much better. There is now a section for "deductions", and you can put your planned donations there. The IRS has an [online calculator](https://apps.irs.gov/app/tax-withholding-estimator/) that is pretty good.

Since you normally only fill this out when you start a new job, if you're making a big change to your taxes, like a large increase in donations, you should ask your employer for a new W4 to fill out.

If your state also has an income tax (most states) then it likely has a different form, (ex: in MA it's [Form M-4](https://www.mass.gov/doc/form-m-4-massachusetts-employees-withholding-exemption-certificate/download), in CA it's [DE 4](https://edd.ca.gov/siteassets/files/pdf_pub_ctr/de4.pdf)), and unfortunately they generally still use an "allowances" system. If your state doesn't allow you to deduct donations (ex: MA and NJ) these forms are relatively easy to figure out because you just put down your information like anyone else. If your state does (ex: CA, NY), it probably has a much lower limit than the federal government ([table](https://www.uscharitablegifttrust.org/tax-treatment-of-charitable-contributions.php)), so when following the instructions only consider the amount that you plan to donate that your state will count.

Overall, if you've done this correctly, when it's time to file taxes next year you should be looking at a small bill/refund. If you are off, however, as long as you adjust for next year you should be fine: the [rules for underpayment](https://www.irs.gov/pub/irs-pdf/i2210.pdf) are relatively friendly as the IRS goes.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100248240650582)*
# A Small Negative Result on Debate

Some context for this [new arXiv paper](https://arxiv.org/abs/2204.05212) from my group at NYU:

*   We're working toward [sandwiching](https://www.alignmentforum.org/posts/PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models) experiments using our [QuALITY long-document QA dataset](https://arxiv.org/abs/2112.08608), with reading time playing the role of the the expertise variable. Roughly: Is there some way to get humans to reliably answer hard reading-comprehensions questions about a ~5k-word text, without *ever* having the participants or any other annotators take the ~20 minutes that it would require to actually read the text.
*   This is an early writeup of some negative results. It's earlier in the project that I would usually write something like this up, but some authors had constraints that made it worthwhile, so I'm sharing what we have.
*   Here, we tried to find out if *single-turn* debate leads to reliable question answering: If we give people high-quality arguments for and against each (multiple-choice) answer choices, supported by pointers to key quotes in the source text, can they reliably answer the questions under a time limit?
*   We did this initial experiment in an oracle setting; We had (well-incentivized, skilled) humans write the arguments, rather than an LM. Given the limits of current LMs on long texts, we expect this to give us more information about whether this research direction is going anywhere.
*   It didn't really work: Our human annotators answered at the same low accuracy with and without the arguments. The selected pointers to key quotes did help a bit, though.
*   We're planning to keep pursuing the general strategy, with multi-turn debate—where debaters can rebut one another's arguments and evidence—as the immediate next step.
*   Overall, I take this as a *very* slight update in the direction that debate is difficult to use in practice as an alignment strategy. Slight enough that this probably shouldn't change your view of debate unless you were, for some reason, interested in this exact constrained/trivial application of it.
# Does the rationalist community have a membership funnel?

I was talking to someone the other day about the ways in which I've noticed the \[Berkeley\] rationalist community changing. One of the main ways was that group houses seemed to be disappearing. People were getting older, moving away, or just moving into their own houses to have kids. It then occurred to me that it doesn't seem like this is happening with the EA side of the community. Thinking about it more, it seems to me that EA has a quite strong funnel in the form of student groups. I semi-regularly hear about events, companies, projects, or just impressive people that are coming from EA student groups. Meanwhile I'm not even aware of a rationalist student group (although I'm sure there are some).

When I think about where rationalists came from, my answer is 1) EY writing the original sequences, and 2) EY writing HPMOR. It feels like those things happened, tons of people joined, and then they stopped happening, and people stopped joining. Then people got older, and now we have a population pyramid problem.

I think this is something of a problem for the mission of preventing AI x-risk. It is of course great to have lots of EAs around, but I think that people that the rationalist community would differentially appeal to would provide a lot of value that EA-learning people would be a lot less likely to provide (focus on AI, obsessive investigation into understanding confusing yet important subjects, etc.).

Do others agree with the pattern? Do you also see it as a problem? Any suggestions for what we could do about it? Why aren't there many rationalist student groups?
# “Fragility of Value” vs. LLMs

It seems like we keep getting LLMs that are better and better at *getting* the point of fairly abstract concepts (e.g. understanding jokes). As compute increases and their performance improves, it seems increasingly likely that human “values” are within the class of not-that-heavily-finetuned LLMs.

For example, if I prompted a GPT-5 model fine-tuned on lots of moral opinions about stuff: “[details of world], would a human say that was a more beautiful world than today, and why?” I… don’t think it’d do terribly?

The same goes for e.g. how the AI would answer the trolley problem. I’d guess it’d look roughly like humans’ responses: messy, slightly different depending on the circumstance, but not genuinely orthogonal to most humans’ values.

This is obviously vulnerable to adversarial examples or extreme OOD settings, but then robustness seems to be increasing with compute used, and we can do a decent job of OOD-catching.

Is there a modern reformulation of “fragility of value” that addresses this obvious situational improvement? Because as of now, the pure "Fragility of Value" thesis seems a little absurd (though I’d still believe a weaker version).
# Common Knowledge is a Circle Game for Toddlers

I wanted to jot down a particular instance of a common interaction I have with my daughter.

My child, wife, and I are sitting around the coffee table in the living room discussing the plan to take the car to the mechanic the next day. We describe this as *going on an adventure* to the child as a lingering holdover from isolation during covid, which seems to be fine because she enjoys car rides. What is different from the usual this time is that I am coming; normally car adventures are my wife driving, because she runs most of the errands. The following (simplified) dialogue takes place:

Child: "Dad, can you come in the car tomorrow?"

Me: "I *am* coming in the car tomorrow."

Child: "Really?! You are coming?"

Me: "Yes."

Child: "Mom! Did you know Dad is coming in the car tomorrow?"

Mom: "Yes! He is coming with us tomorrow."

Child: "Dad! Mom knows you are coming tomorrow! You are both coming!"

Me: "Yes!"

Keep in mind we have all been present for the whole conversation. We are sitting around a coffee table. None of us are more than three feet from each other.

It struck me that what my daughter was doing was trying to explicitly track what her mom and I knew. I expect this falls somewhere under the theory-of-mind stage of development, but what stuck out to me was the procedure she undertook, which looked something like three loops:

1.  One loop with me, of establishing the fact and confirming the fact.
2.  A lateral loop to mom, of establishing that mom knows the fact and confirming that mom knows the fact.
3.  A lateral loop back to me, of establishing that I know that mom knows, and confirming that I know that mom knows.

The looping strategy reminded me of [Circle Games](https://www.lesswrong.com/posts/7Fft9kNTvqixttGc3/circle-games), and only after this did it occur to me that the *object* of the game was [common knowledge](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge).

My child has been playing these kinds of games for a while now, on reflection; but recently they have grown in frequency and elaboration enough that I began to wonder why she seemed to be spending more time on what I instinctively treat as mundane details. The connection to common knowledge did not crystalize until this event.
# What to include in a guest lecture on existential risks from AI?

A professor I'm friendly with has been teaching a course on AI ethics this semester, and he asked me if I could come give a guest lecture on "AI apocalypse" scenarios. What should I include in the lecture?

Details:

*   Audience is mostly graduate engineering students. Some are in a program focused on AI, others are in a systems engineering program and don't otherwise have much knowledge about AI beyond this course.
*   This will be I think the final class for the semester. Previous classes have covered topics like privacy, bias, explainability, social coordination failures (including race to the bottom dynamics), self-driving cars, AI in social media, lethal autonomous weapons, AI in healthcare, implementing ethics in organizations, and the economic impacts of AI. I did not attend those lectures so I don't know exact details of what was or wasn't covered.
*   I have leeway to include pretty much anything I want in the class and to focus on any topic(s) I want. The professor would like me to include something about forecasting, but that's optional.
*   I can optionally assign something short as an assignment before class, but it is the end of the semester so nothing more than that. There will not be any homework based on the class, I think.
*   The class will be 2 hours long.
*   I do not have a huge amount of time to prepare for the lecture - it's relatively soon (April 27) and I have multiple other work / school / family obligations I need to attend to between now and then.

If anybody has relevant material I could use, such as slides or activities, that would be great! Also, if anybody wants to help develop the material for this class, please message me (preferably at my work email - [Aryeh.Englander@jhuapl.edu](mailto:Aryeh.Englander@jhuapl.edu)).

As a bonus, I expect that material for a class of this sort may turn out to be useful for plenty of other people on this and related forums, either for themselves or as a tool they can use when presenting the same topic to others.

*\[Note: I am posting this here with permission from the professor.\]*
# Emacs Recentering With Context

In emacs, when you press Ctrl-L, by default it runs `[recenter-top-bottom](https://www.gnu.org/software/emacs/manual/html_node/emacs/Recentering.html)`. This scrolls the document ("buffer") so the line the cursor is currently on is centered in the screen. If you press it again, it scrolls so the cursor is at the very top of the screen, and again puts the cursor at the very bottom. I've always been mildly irritated by this: the middle of the screen is lower than I want for the initial state; I generally want more context below the cursor than above. Additionally, scrolling to the top or bottom is nearly useless because I almost always want at least a little bit of context on both sides of the cursor.

Everything in emacs *can* be configured, but sometimes it's quite a pain, so I kept putting off looking into this. With Nora napping and the older two playing with a neighbor friend, however, I finally looked into this, and it turns out to be very easy. There's a `recenter-positions` variable that defaults to `'(middle top bottom)`, but you can set it to anything you want. The default is equivalent to `'(0.0, 0.5, 1.0)`, and I'm now using:

(setq recenter-positions '(0.25 0.5 0.9))

This means that the first Ctrl-L puts my cursor 25% of the way down the page, which is where I'm most likely to want it. If I find I want more context above I can press it again, to put it at 50%, and again to put it at 90%.

These are good for my usage, but I run [tall terminals](https://www.jefftk.com/p/terminal-preferences) and if I ran shorter ones I might want different values.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100248327501532)*