Harley-ml/lesswrong huggingface dataset:
# How would we check if "Mathematicians are generally more Law Abiding?"

In [Local Validity](https://www.lesswrong.com/posts/WQFioaudEH8R7fyhm/local-validity-as-a-key-to-sanity-and-civilization), Eliezer notes:

> But I would venture a guess and hypothesis that you are better off buying a used car from a random mathematician than a random non-mathematician, even after controlling for IQ. The reasoning being that mathematicians are people whose sense of Law was strong enough to be appropriated for proofs, and that this will correlate, if imperfectly, with mathematicians abiding by what they see as The Law in other places as well. I could be wrong, and would be interested in seeing the results of any study like this if it were ever done. (But no studies on self-reports of criminal behavior, please. Unless there's some reason to believe that the self-report metric isn't measuring "honesty times criminality" rather than "criminality".)

I'm guessing such a study hasn't been done, but it seems like the sort of thing you should be able to actually go and check. 

I'm interested in both:

*   What is the expensive, impractical study you could hypothetically run that would give strong evidence about this question?
*   What's the cheapest, practical study we could  run that would provide at least some meaningful data about it?
# Update on Ought's experiments on factored evaluation of arguments

[Ought](https://ought.org/) has written a detailed update and analysis of recent experiments on factored cognition. These are experiments with human participants and don’t involve any machine learning. The goal is to learn about the viability of [IDA](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd?_ga=2.44167719.2055190071.1578702594-1142780176.1552454685), [Debate](https://openai.com/blog/debate/), and related [approaches](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) to AI alignment. For background, here are some prior LW posts on Ought: [Ought: Why it Matters and How to Help](https://www.lesswrong.com/posts/cpewqG3MjnKJpCr7E/ought-why-it-matters-and-ways-to-help)), [Factored Cognition presentation](https://www.lesswrong.com/posts/DFkGStzvj3jgXibFG/factored-cognition).

Here is the opening of the research update:

> **Evaluating Arguments One Step at a Time**

> We’re studying [factored cognition](https://ought.org/research/factored-cognition): under what conditions can a group of people accomplish complex cognitive tasks if each person only has minimal context?

> In a recent experiment, we focused on dividing up the task of evaluating arguments. We created short, structured arguments for claims about movie reviews. We then tried to distinguish valid from invalid arguments by showing each participant only one step of the argument, not the review or the other steps.

> In this experiment, we found that:

> 1\. Factored evaluation of arguments can distinguish some valid from invalid arguments by identifying implausible steps in arguments for false claims.

> 2\. However, experiment participants disagreed a lot about whether steps were valid or invalid. This method is therefore brittle in its current form, even for arguments which only have 1–5 steps.

> 3\. More diverse argument and evidence types (besides direct quotes from the text), larger trees, and different participant guidelines should improve results.

> In this technical progress update, we describe these findings in depth.

The rest of the post is [here](https://ought.org/updates/2020-01-11-arguments).
# How has the cost of clothing insulation changed since 1970 in the USA?

I sometimes hear the claim that innovation in the physical world has stagnated since around 1970. More specifically, chapter 1 of The Rise and Fall of American Growth by Robert J Gordon claims that there has been basically no innovation in clothing other than changes in fashion. This is somewhat contrary to my intuition (although I definitely believe that innovation in the 50 years before 1970 was greater that in the 50 years after), and price of insulation seems like a relatively objective metric for this.

My favourite type of response would be time series data of [clo](https://en.m.wikipedia.org/wiki/Clothing_insulation?wprov=sfla1) per inflation-adjusted dollar, but I'd also appreciate people's subjective experience of this.
# Predictors exist: CDT going bonkers... forever

I've been wanting to get a better example of CDT ([causal decision theory](https://en.wikipedia.org/wiki/Causal_decision_theory)) misbehaving, where the behaviour is more clearly suboptimal than it is in the [Newcomb problem](https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality) (which many people don't seem to accept as CDT being suboptimal), and simpler to grasp than [Death in Damascus](https://arxiv.org/pdf/1710.05060.pdf).

## The "predictors exist" problem

So consider this simple example: the player is playing against Omega, who will predict their actions[^predtime]. The player can take three actions: "zero", "one", or "leave".

[^predtime]: Omega will make this prediction not necessarily before the player takes their action, not even necessarily without seeing this action, but still makes the prediction independently of this knowledge. And that's enough for CDT.

If ever they do "leave", then the experiment is over and they leave. If they choose "zero" or "one", then Omega will predict their action, and compare this to their actual action. If the two match, then the player loses $1$ utility and the game repeats; if the action and the prediction differs, then the player gains $3$ utility and the experiment ends.

Assume that actually Omega is a perfect or quasi-perfect predictor, with a good model of the player. An [FDT](https://arxiv.org/abs/1710.05060) or EDT agent would soon realise that they couldn't trick Omega, after a few tries, and would quickly end the game.

But the CDT player would be incapable of reaching this reasoning. Whatever distribution they compute over Omega's prediction, they will always estimate that they (the CDT player) have at least a $50\%$ chance of choosing the other option[^choose], for an expected utility gain of at least $0.5(3) + 0.5(-1)=1$.

[^choose]: For example, suppose the CDT agent estimates the prediction will be "zero" with probability $p$, and "one" with probability 1-p. Then if $p \geq 1/2$, they can say "one", and have a probability $p\geq 1/2$ of winning, in their own view. If $p<1/2$, they can say "zero", and have a subjective probability $1-p > 1/2$ of winning.

Basically, the CDT agent can never learn that Omega is a good predictor of themselves[^other]. And so they will continue playing, and continue losing... for ever.

[^other]: The CDT agent has no problem believing that Omega is a perfect predictor of *other agents*, however.
# Is backwards causation necessarily absurd?

In [Newcomb's problem](https://wiki.lesswrong.com/wiki/Newcomb's_problem) an agent picks either one-box or two-box and finds that no matter which option they picked, a predictor predicted them in advance. I've gone to a lot of effort to explain how this can be without requiring backwards causation ([The Prediction Problem](https://www.lesswrong.com/posts/YpdTSt4kRnuSkn63c/the-prediction-problem-a-variant-on-newcomb-s), [Deconfusing Logical Counterfactuals](https://www.lesswrong.com/posts/BRuWm4GxcTNPn4XDX/deconfusing-logical-counterfactuals)), yet now I find myself wondering if backwards causation is such a bad explanation after all.

Unfortunately I'm not a physicist, so take what I say with a grain of salt, but I'll sketch out some reasons why backwards causation might not be as ridiculous as it first seems and hopefully someone else develops this in more detail.

One prominent theory of time is [Eternalism](https://www.wikiwand.com/en/Eternalism_(philosophy_of_time)) in which there is no objective flow of time and terms like "past", "present" and "future" can only be used in a relative sense. An argument in favour of this is that it is often very convenient in physics to model space-time as a 4-dimensional space. If time is just another dimension, why should the future be treated differently than the past? Nothing in this model differentiates the two. If we have two blocks X and Y next to each other, we can view either X as the left one or Y as the left one depending on the direction we look at it from. Similarly, if A causes B in the traditional forwards sense, why can't we symmetrically view B as backwards causing A, where again if we viewed it another way A to B would be backwards causation and B to A would be forwards causation.

Another relativistic argument against time flowing is that simultaneity is only defined relative to a reference frame. Therefore, there is no unified present which is supposed to be what is flowing. This doesn't mean that the universe couldn't be described by a unidirectional graph. However, it does greatly undermine any trust in our naive intutions related to time.

Thirdly, entropy has often been the arrow of time with other physical laws claimed to be reversible. We are in a low-entropy world so entropy increases. However, if we were in a high-entropy world, it would decrease, so time and causation would seem to be going backwards (from our perspective). This would seem to suggest that backwards causation is just as valid a phenomenon as backward causation.

I want finish by reminding readers again that I am not a physicist. This post is more intended to spark discussion that anything else.

(Another possibility I haven't discussed is that causation might be in the map rather than the territory)
# Remote AI alignment writing group seeking new members

If you're interested, then email me (<given name of President
Nixon>.moehn@posteo.de) or PM me on LessWrong.

We meet weekly for 15 min ⨉ number of members. At the moment the meetings consist of explaining aspects of our
research to each other. This helps us get better at explaining (sometimes off
the cuff), as well following technical explanations and coming up with questions
about them. Also, questions from others are a good way to find holes in one’s
understanding and patch them. – Better find them among friends and patch them
early than getting flustered in public later.

The research topics of the two current members are:

- impact measures/instrumental convergence à la Alex Turner
- iterated distillation and amplification à la Paul Christiano

Even if your topic is far afield, we'll try and make it work. Explanations and questions are universal.

There used to be four members. This means:

1. The meetings are useful for some and not for others. If you're unsure, I
   suggest you join. If you don't get enough value for your time, you can leave again. We
   respect that.

2. Two spots are free!

See also [the original
announcement](https://www.lesswrong.com/posts/zXnFhQsYbHwLePPqS/looking-for-remote-writing-partners-for-ai-alignment).
# Political Roko's basilisk

Why has there never been a "political Roko's basilisk", i.e. a bill or law that promises to punish any member of parliament who voted against it (or more generally any individual with government power, e.g. judge or bureaucrat, who did not do everything in their capacity to make it law)?

Even if unconstitutionality is an issue, it seems like the "more general" condition would prevent judges from overturning it, etc. And surely there are countries with all-powerful parliaments.
# Why Do You Keep Having This Problem?

One thing I've noticed recently is that when someone complains about how a certain issue "just keeps happening" or they "keep having to deal with it", it often seems to indicate an unsolved problem that people may not be aware of. Some examples:

*   Players of a game repeatedly ask the same rules questions to the judges at an event. This doesn't mean everyone is bad at reading -- it likely indicates an area of the rules that is unclear or misleadingly written.
*   People keep trying to open a door the wrong way, either pulling on a door that's supposed to be pushed or pushing a door that's supposed to be pulled -- it's quite possible the handle has been designed poorly in a way that gives people the wrong idea of how to use it. ([The Design of Everyday Things](https://smile.amazon.com/Design-Everyday-Things-Revised-Expanded/dp/B07L5Y9HND/ref=sr_1_1?keywords=design+of+everyday+things&qid=1578965322&sr=8-1) has more examples of this sort of issue.)
*   Someone keeps hearing the same type of complaint or having the same conversation about a particular policy at work -- this might be a sign that that policy might have issues. \[1\]
*   Every time someone tries to moderate a forum they run, lots of users protest against their actions and call it unjust; this might be a sign that they're making bad moderation decisions.

I'm not going to say that _all_ such cases are ones where things should change -- it's certainly possible that one might have to take unpopular but necessary measures under some circumstances -- but I do think that this sort of thing should be a pretty clear warning sign that things might be going wrong.

Thus, I suspect you should consider these sorts of patterns not just as "some funny thing that keeps happening" or whatever, but rather as potential indicators of "bugs" to be corrected!

  

\[1\] This post was primarily inspired by a situation in which I saw someone write "This is the fifth time I've had this conversation in the last 24 hours and I'm sick of it" or words to that effect -- the reason they had kept having that conversation, at least in my view, was because they were implementing a bad policy and people kept questioning them on it (with perhaps varying degrees of politeness).
# Terms & literature for purposely lossy communication

Say I have a 10 of coins that could either be red or blue, and heads or tails. This has 20 bits of information.

My friend is interested in knowing about which of them are heads or tails, but doesn’t care about the color. I then decide to only tell my friend the heads/tails information; so, 10 bits of information.

Another example:
With image compression, there's a big difference between "Accurately reconstruct this image, pixel by pixel" vs. "Accurately reconstruct what the viewer would remember, which is basically just 'Lion on grass'".

I’d feel uncomfortable calling this translation “compression”, because there was definitely intentional information loss. Most of the literature on compression is about optimally maintaining information, not on optimally loosing it.

Are there other good terms or literature for this?
# New paper: The Incentives that Shape Behaviour

Abstract: Which variables does an agent have an incentive to control with its decision, and which variables does it have an incentive to respond to? We formalise these incentives, and demonstrate unique graphical criteria for detecting them in any single decision causal influence diagram. To this end, we introduce structural causal influence models, a hybrid of the influence diagram and structural causal model frameworks. Finally, we illustrate how these incentives predict agent incentives in both fairness and AI safety applications.
# Theory of Causal Models with Dynamic Structure?

I'm looking for any work done involving causal models with dynamic structure, i.e. some variables in the model determine the structure of other parts of the model.

I know some probabilistic programming languages support dynamic structure (e.g. the [pyro docs](http://docs.pyro.ai/en/stable/) mention it directly at one point). And of course one can always just embed the dynamic structure in a static structure (i.e. a model of any general-purpose computer), although that's messy enough that I'd expect it to create other problems.

I haven't found much by quick googling (too many irrelevant things use similar terms) so I'd appreciate any pointers at all. At this point I've found basically-zero directly relevant work other than PPLs, and I don't even know of any standard notation.
# Litany Against Anger

The map is not the territory, \
the word is not the thing. 

My anger is not the trigger, \
triggers do not become more or less important \
as I feel anger wax and wane. 

Emotions are motivation, \
they're supposed to get me to do the right thing. 

Defuse from my frustration, \
breathe and wait for clarity calm will bring.
# Are the bad epistemic conditions global?

You've set this in the context of USA politics and society. Clearly it is not just limited to the USA. I seem to recall similar types of things occurring in Europe over the past 20 - 30 years.

I also suspect one might find this type of problem more prevalent within the environmental discussions and various camps.

Last, the the behavior of a lot of mainland Chinese students in other countries seems to be applying these types of tactics in the context of discussions and positive views about the Hong Kong protests and pro-democracy movements.

It does seem that some pendulum has been swinging in this direction for some time everywhere. So perhaps it is actually a normal social/political cycle type event we're seeing play out. But that does still beg the question about underlying forces and perhaps more importantly the countervailing forces that might return us to more reasoned and tolerant dialogue when differences exist.
# The Bentham Prize at Metaculus

[Metaculus](https://www.metaculus.com/questions/?invite=DW5XlK) is running a small competition right now called [the Bentham Prize](https://www.metaculus.com/questions/3523/announcing-the-bentham-prize/?invite=DW5XlK). The premise is simple: contribute to the discussions of [questions on Animal Welfare](https://www.metaculus.com/questions/?search=cat:series--animal-welfare?invite=DW5XlK). The best record of contributions (subjectively assessed by human judges) will win.

If you've been waiting for a reason to start practicing forecasting, this seems as good as any. It's gotten off to a slow start (in part because the Metaculus community has become [quite distracted by another matter](https://www.metaculus.com/questions/?search=cat:bio--infectious-disease--novel-coronavirus?invite=DW5XlK)), so the field is wide open!
# What research has been done on the altruistic impact of the usual good actions?

What I mean by the "usual good actions" is notable good deeds you do may be a few times every week. Some examples:

*   Help a friend end an abusive relationship.
*   Go out of your way to help a stranger.
*   Do something extra nice and thoughtful for your partner.
*   Show up in person to support your friend's endeavor.
*   Procure a Burning Man ticket for your friend.
*   Refrain from calling your co-founder's idea "the dumbest thing I've heard this year" and do your best to listen.

What I mean by "altruistic impact" is probably QALYs, but I guess it's harder to use that metric here because you're working with changes in a pretty high quality life already vs. the difference between alive and dead. So may be there's a better metric. I'm also interested in the indirect impact of these actions (see my [comment](https://www.lesswrong.com/posts/ZRegXDB5BQEhPXY4W/what-research-has-been-done-on-the-altruistic-impact-of-the?commentId=oMWxWcue753MNKxcq)).

I'm also open to better a rephrasing of this question.

Oh, and I'm happy to hear everyone's thoughts on this without research too.
# Rationalist prepper thread

There are different estimates of the possible severity of the current coronavirus outbreak. One estimation is based on the straight extrapolation of the exponential growth of the infected people number, which doubles every two days. This implies that the whole population of Earth will be ill in March. Another view takes into account that many mild cases are not included in the stat, so lethality is small and probably not everybody will be ill at all. We just don't know yet.

How we should act personally in this situation?

Firstly, we should act in the way, which should be good if everybody will act in the same way. For example, panic evacuation is not good, as if everybody starts running away, this will only increase the dissemination of the virus. Thus the best approach is home self-isolation, lowering of the outside contacts, wearing protecting gear (mask, goggles, gloves), hand cleaning. UV-lamps may help in the disinfection of things.

It is also good to invest in improving ones’ immune system by health food, vitamins, light therapy, as it is our best protection of the virus. Evacuation into a cold county house would weaken the immune system.

It is probably reasonable to collect one month supply of food, water, and other essentials, as the possible pandemic may affect food manufacturing, and also wholesale buying minimise future need to go outside.
# Algorithms vs Compute

Two scenarios:

*   I take a vision or language model which was cutting edge in 2000, and run it with a similar amount of compute/data to what's typically used today.
*   I take a modern vision or language model, calculate how much money it costs to train, estimate the amount of compute I could have bought for that much money in 2000, then train it with that much compute

In both cases, assume that the number of parameters is scaled to available compute as needed (if possible), and we generally adjust the code to reflect scalability requirements (while keeping the algorithm itself the same).

Which of the two would perform better?

CLARIFICATION: my goal here is to compare the relative importance of insights vs compute. "More compute is actually really important" is itself an insight, which is why the modern-algorithm scenario talks about compute cost, rather than amount of compute actually used in 2000. Likewise, for the 2000-algorithm scenario, it's important that the model only leverage insights which were already known in 2000.
# Survival and Flourishing grant applications open until March 7th ($0.8MM-$1.5MM planned for dispersal)

The Survival and Flourishing Fund has taken over most of the grantmaking of the Berkeley Existential Risk Initiative (BERI) and just opened up another grant round (after the [last round](https://forum.effectivealtruism.org/posts/ctc2PE7Yu7WF3dKeN/survival-and-flourishing-fund-grant-applications-open-until) in September):

[http://survivalandflourishing.fund/#2020-h1-grant-round](http://survivalandflourishing.fund/#2020-h1-grant-round)

The plan is to make a total of $0.8MM-$1.5MM in grants to organizations working on the long term flourishing and survival of humanity.

At this point in time SFF can only make grants to charities and not individuals, so if you have a project or organization that you want to get funding for, you will have to either already be part of an established charity, found one, or be sponsored by an existing charity. We have some initial plans to run a grant round for more project and individual-based grants, but nothing concrete yet.

Also happy to again answer any questions you have in the comments, to the best of my knowledge.
# Mod Notice about Election Discussion

*TL;DR: Posts discussing the presidential election may be hidden, not appearing in recent discussion, on the home page, or on the allPosts page.*

Ben Landau-Taylor wrote [The Four Year Locusts](https://benlandautaylor.com/2018/09/22/the-four-year-locusts/) to describe the periodic problem of politics consuming everything within reach. From it:

> While the swarm is too big to defeat entirely, local defense is possible. Some areas can be kept clear. What is precious can be guarded. The swarm is temporary; you need only outlast it.

Even in the best of times, Less Wrong discourages posts on politics; [Politics is the Mind Killer](https://www.lesswrong.com/posts/9weLK2AJ9JEt2Tt8f/politics-is-the-mind-killer), as the saying goes, a post that turned into a norm a bit more expansive than the original post suggests. We are not currently in the best of times.

I see three main reasons why this matters:

1.  Object-level harms to the discourse from using political examples. It's both harder for people to discuss politics, and harder for them to agree on [the right abstractions](https://www.lesswrong.com/posts/bFv8soRx6HB94p5Pg/against-premature-abstraction-of-political-issues). If you discuss the abstractions directly, you can avoid those issues.
2.  Pull factors for the wrong crowd. Suppose some people care 100% about rationality, and discuss rationality things; people who care 80% about rationality and 20% about politics join in and talk about rationality. But if the people who care a little about politics start discussing politics with each other, then they pull in people who care 60% about rationality, and 40% about politics, and perhaps things spiral out of control, especially if it's somehow easier or more rewarding to talk about politics.
3.  Push factors for the right crowd. Suppose people want to avoid politics, because of disinterest, reputation harms, or the increased surface area for enemy action. Then if your forum becomes about that thing, people who would otherwise be interested in your thing will leave, or [be harassed](https://slatestarcodex.com/2019/02/22/rip-culture-war-thread/).

So back to the locusts. How to respond? People like Bryan Caplan take a [social media hiatus](https://www.econlib.org/my-social-media-hiatus/). Here on LW, we're going to try to minimize the visibility of political discussion, especially with regards to presidential politics. We're interested in raising the sanity waterline more broadly, but also in picking our battles, and presidential politics is a battle we definitely are not interested in picking.

We're considering several different ways of minimizing visibility, and will deploy them as they seem necessary. You'll still be able to write personal blog posts on topics as you see fit with the normal restrictions, but those posts might not show up on the main page and their discussion might not show up in recent discussion. (If you want to make sure you see all posts by another user, subscribe to their posts by clicking on their username and then hitting the "subscribe to posts" link.)
# TAISU - Technical AI Safety Unconference

**COVID-19 Update!**

TAISU in it's planed form is cancelled. But there will be a **Web-TAISU** around the same theme, and around the same time. I will make an announcement and probably open up for more applications when this thing is a bit more planed out.

* * *

**Start:** Thursday, May 14th, 10am  
**End:** Sunday, May 17th, 7pm  
**Location:** [CEEALAR \[formerly the EA Hotel\]](http://ceealar.org/wiki/#travel), 36 York Street, Blackpool, UK  
  
I ran this unconference in August last year. Since that event was a success, I decided to do another one.

This is an unconference around the question: “How do we design AI such that it is safe in the long run?” This includes things like **alignment, corrigibility, transparency** and also **understanding humans** and **human value structures** better. Questions regarding governance and coordination are not part of this particular event.

Because this is an unconference, it will become what you make of it. There will be an empty schedule which you, the participants, will fill up with talks, discussions and more.

You don’t have to stay the full length of the unconference. It is possible to join as many or as few days as you want to, and you are also welcome to stay longer at CEEALAR before or after the unconference.

**Who can participate**  
To be able to have a high level of discussion during the unconference, I'll require that all participants have some prior involvement with AI Safety. This year I’m also reserving at least half the spots for people who are currently working on AI Safety (as opposed to, have done so in the past, or are studying to do so in the future).

**Applications**  
If this seems like something for you, fill out this [application form](https://docs.google.com/forms/d/e/1FAIpQLSdLLvjGzvIPi-iXbYzT3oTVlTU-kZkxI09wYq9EVMJvig6ZyQ/viewform). Application deadline is February 23rd. I will inform you who is accepted on February 28th at the latest (If you need to know sooner, let me know). If there is still room after that I will accept qualified participants on a first come first serve basis.

_**Update (Feb 24):** TAISU is not full yet. I will now accept people on a first come first serve bases, for anyone who is qualified._

**Price**  
£150 for the unconference, regardless of how many days you are attending.  
If this is more than you can afford, let me know. If I think you belong at the unconference, I’ll give you as much discount as you need.

**Food:** All meals will be provided by CEEALAR, no extra charge.

**Accommodation**  
You are responsible to book and pay for your own accommodation.

You can book a bed at EA Hotel there through their [booking system](https://ceealar.org/booking/). Choose the “Grant Application (working on EA projects)” option and let them know you are there for the TAISU. The stay at the hotel is pay what you want, but it will be appreciated if you pay £12/night, which is the cost price.

Alternatively you can get a room at any of the many [nearby hotels](https://docs.google.com/document/d/1Z8HxWa8kp2wHKqlgPaIoYv8EinxQXp79-Z7Xif-eKSY/edit?usp=sharing).
# Slide deck: Introduction to AI Safety

This is from a presentation that I've been giving over the past few months to various groups within the Johns Hopkins University Applied Physics Laboratory (APL), where I work. I'd be interested to hear people's comments, critiques, suggestions, etc.

Note that this is a somewhat older version of the presentation, and I've made several improvements since then. But this is the version that finally made its way through our Office of Technology Transfer and was approved for public release. At some point I'll submit an updated version for public release approval, but at the moment this is the version I can share.
# Artificial Intelligence, Values and Alignment

I haven't yet had a chance to read the whole thing, but DeepMind has a [new paper](https://deepmind.com/research/publications/Artificial-Intelligence-Values-and-Alignment) up from Iason Gabriel titled "Artificial Intelligence, Values and Alignment". Here's the abstract:

> This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify 'true' moral principles for AI; rather, it is to identify fair principles for alignment, that receive reflective endorsement despite widespread variation in people's moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified.

As I said, I haven't read the [whole thing](https://arxiv.org/abs/2001.09768), but I'm somewhat suspicious of moving in the direction of principles as discussed in this paper since, based on my skimming, it seems to be suggesting we pick the method of aggregating human preferences an aligned AI would use with a higher level of specificity than [I suspect is safe](https://arxiv.org/abs/1807.09836) (given the level of confidence we would need to have to make a particular choice).

Nonetheless this paper looks interesting and covers a number of topics in detail I haven't seen show up much in the academic literature even if we talk about them here, so I look forward to looking at this paper more closely and possibly having more detailed comments to offer then.
# how has this forum changed your life? 

Hello! I am totally new here so please bear that in mind in the event I commit faux pas! I'm a writer who has written a LOT about rationality and when rationality eludes us. These are books like So You've Been Publicly Shamed, Them, The Psychopath Test and The Men Who Stare At Goats, among others.

I don't know if this question has been asked elsewhere, but I'd love to know: has learning to be more rational impacted your everyday lives, in small or (perhaps more interestingly) BIG ways? Have there been occasions when you've put these skills (perhaps learned at a workshop) into practice in your domestic or work lives and the ripples have been surprising/consequential in positive or negative ways?

I'm thinking of writing about what happens when a person learns to be more rational, hence my question.

Hope this is the start of a wonderful thread!

Jon Ronson
# Existing work on creating terminology & names?

I'm in the process of coming up with terminology for various theories, similar to lots of other work on LessWrong and The EA Forum.

Naming things is a bit of a unilateralist action. While community members don't have to accept a specific naming proposal, they are likely to do so if they like the concept. I can't think of many cases where Eliezer or someone named a concept, and the community decided that that name was poor, and renamed it.

However, I can't find much theory on how to figure out great names for things, or even what to consider when doing so. I would have expected there to be comprehensive discussion around Information Architecture, UX Design, or the Library Sciences on this topic, but couldn't identify much outside of card sorting and a few lists of rough heuristics.

This was also an issue for me when I did more software engineering, and I was then also frustrated by the lack of discussion I could find. The best there was work on Software Patterns, which I used primarily for naming conventions.

Some related links I could find:  
https://en.wikipedia.org/wiki/Naming_convention  
https://www2.staffingindustry.com/eng/Editorial/Archived-Blog-Posts/Adam-Pode-s-Blog/Probably-the-best-file-naming-convention-ever  
https://www.invisionapp.com/inside-design/naming-conventions/  
https://ux.stackexchange.com/questions/48578/naming-features-of-an-app-or-site
https://www.martyneumeier.com/strong-vs-weak-names
# REVISED: A drowning child is hard to find

Substantial revisions to clarify the post's core claim, including but not limited to this summary at the end:

> # Summary
> 
> * Effective Altruism claims that there is a large funding gap for cheap well-understood developing-world interventions.
> * Even the most aggressive plausible construal of this claim implies an annual funding gap that could be covered completely by existing major institutional donors.
> * If this is true, it implies opportunities for comparatively cheap experiments (relative to the endowments of major donors in the space) with extremely high information value.
> * Such experiments have not happened either because they are impossible, or because the relevant institutional donors think they have better things to do with their money.
> * Neither scenario suggests that small donors should try to fill this funding gap. If they trust big donors, they should just give to the big donors. If they don't, why should they believe a story clearly meant to extract money from them?

[Original linkpost](https://www.lesswrong.com/posts/qaYeQnSYotCHQcPh8/drowning-children-are-rare)
# Instrumental Occam?

In _[The Real Rules have No Exceptions](https://www.lesswrong.com/posts/duxy4Hby5qMsv42i8/the-real-rules-have-no-exceptions),_ Said says (apparently speaking of _instrumental_ rules, not epistemic ones):

> _Prefer simplicity in your rules. Be vigilant that your rules do not grow too complex; make sure you are not relaxing the legitimacy criteria of your exceptions. Periodically audit your rules, inspecting them for complexity; try to formulate simpler versions of complex rules._

This is a very plausible principle. I have had similar thoughts myself. The idea seems to have merit in practice. But is there any theoretical analogue?

Minimum description length (Occam's Razor) is an epistemic principle. In Bayesian terms, probabilities must sum to one, which translates information-theoretically (via optimal encodings) to the idea that there are only so many short codes, so we have to assign them carefully to the most probable things.

However, expected utility is not a limited resource in this way. Updating to think an option is better than previously thought [doesn't necessarily make other options worse](https://www.lesswrong.com/posts/XYCEB9roxEBfgjfxs/the-scales-of-justice-the-notebook-of-rationality). Selecting policies is not so different from selecting raw actions. And for Bayesian decision theory, it doesn't really matter whether a policy is specified as a big table of observation/action pairs, vs an elegantly specified rule.

Optimality cares not for elegance!

Yet, even in the relatively formal world of machine learning, the practice seems contrary to this. When you are optimizing a neural network, you don't actually care that much whether it's something like a hypothesis (making predictions) or something like a policy (carrying out actions). You apply the same kind of regularization either way, as far as I understand (regularization being the machine-learner's generalization of Occam). **(Correction: [this seems not to be the case](https://www.lesswrong.com/posts/qqG2PdZ7pEcM6ev3S/instrumental-occam#gAtCmdrYeFwTtpk2w).)**

We _might_ say that this is because (in some sense) the instrumental uncertainty and the epistemic uncertainty are actually being wrapped up together. But (1) this reply seems overly quick to me at present, and I'd want to understand in detail whether this can be justified; (2) I'm curious if there is a _purely instrumental_ version of Occam to be articulated; it seems intuitively really plausible to me, though technically quite mysterious.

So: is it possible to formulate an instrumental version of Occam? Can we justify a simplicity bias in our policies?
# "Memento Mori", Said The Confessor

## Abstract

The fear of death acts as a sort of master key for introductory rationality concepts. Examining the fear of death ties all the rationality basics together into a coherent framework, including:

* Map/Territory Errors
* Something To Protect
* Keeping Your Identity Small
* Atheism
* X-Risk

[Read more](https://www.thelastrationalist.com/memento-mori-said-the-confessor.html)

----

**Small brain**: Don't think about death. \
**Shining tomagraph**: "After I die I'll go to heaven because I'm a good person." \
**Expanding brain**: "God isn't real, [I find it more comforting to think that this isn't all a test.](https://www.youtube.com/watch?v=4-QAoTXP6pE&t=7m35s)" \
**Galaxy Brain**: *Practice dying*.
# [Link] Beyond the hill: thoughts on ontologies for thinking, essay-completeness and forecasting 

_This is a link-post for: [https://www.foretold.io/c/1bea107b-6a7f-4f39-a599-0a2d285ae101/n/5ceba5ae-60fc-4bd3-93aa-eeb333a15464](https://www.foretold.io/c/1bea107b-6a7f-4f39-a599-0a2d285ae101/n/5ceba5ae-60fc-4bd3-93aa-eeb333a15464)_

_\-\-\-_

_Epistemic status: gesturing at something that feels very important. Based on a true story. Show, don't tell. Release early._

Why are documents and spreadsheets so successful?

Why does code, which is many times more powerful than spreadsheets, have many times fewer users?

I think it's because code not just forces you to express your ideas in code, but also to _think in code_. It imposes constraints on your ontology for thinking.

Having spent the last year working on forecasting, I got some experience with how ontologies can significantly constrain technology projects.

I think such constraints have...

*   heavily limited the usefulness of past forecasting efforts
*   resulted in broad misconceptions about what forecasting could be used for
*   hidden a large space of interesting work that can be unlocked if we solved them

So the link-post is an interactive essay where I attempt to show what solving them might look like in practice, using some technology which is currently not supported on LessWrong.

(Note that the link will not work well on mobile.)
# Money isn't real.  When you donate money to a charity, how does it actually help? 

Money is a shared illusion. Neither paper, gold, nor electron patterns can stop a mosquito or feed a hungry person directly. Transfer of money is a temporary, low friction way to motivate unidentified strangers to preform an action.

So what's the mechanism by which monetary charity works? Are we "just" exploiting a comparative advantage in dollar-gathering in order to force the people on the ground to behave the way we prefer? Do we have any evidence that it can create channels of behavior that are self-sustaining?

On small (and not-so-small, but not universal) margins, money is a fine way to change some behaviors. I'm wondering if there are ways to shift the equilibria at the core rather than the margin.
# Map Of Effective Altruism

In the spirit of my old [map of the rationalist diaspora](https://slatestarcodex.com/2014/09/05/mapmaker-mapmaker-make-me-a-map/), here’s a map of the effective altruist movement:

[![](http://slatestarcodex.com/blog_images/eamap_thumbnail.png)](http://slatestarcodex.com/blog_images/ea_imagemap7.html)

Continents are cause areas; cities are charities or organizations; mountains are individuals. Some things are clickable links with title-text explanations. Thanks to AG for helping me set up the imagemap.
# Looking for books about software engineering as a field 

I work in the software industry but am not a software developer. My job is to write about software development, and I've learned a whole bucketload of terms: stuff like 'linked lists', 'CI/CD', 'performance optimization', 'deploy to AWS', 'dockerize', 'microservices', 'SQL injection', 'multithreaded program', 'vectorized code', and on and on and on. However, a lot of the time I'm basically just Chinese-rooming – I can write about these things, but I don't actually understand how any of them fit together. For example, I've had three people try to explain exactly what an API is to me, for more than two hours total, but I just can't internalize it. I feel that there's some impossible-to-articulate piece I'm missing, and none of the words people say to me about software stuff stick because I'm lacking a foundation on which to build up my understanding.

So my question is, are there any books (or other resources) that explain the field of software engineering as a cohesive whole? I'm not looking for books that will teach me to code, because I don't think that's the thing I want. Feel free to ask clarifying questions. Thanks!

-

EDIT: I realized I should include more context on my work and my background, so here it is:

I have an undergrad degree in physics, which gave me extremely minimal exposure to Python. I also took two quarters of intro CS, one in C and one in Racket. As a result I know how to write a for loop and a bit about very basic algorithms; that's about it. I've been in my current job for nearly a year, and my primary task is to write about the skillsets of individual software engineers. This entails things like connecting someone's verbal knowledge of back-end web development to their experience creating microservices; I can do this quite competently and don't make many technical mistakes. I have also learned a bit on the job regarding a couple data structures, some web stuff, and smatterings of info about ML, data science, DevOps, front-end/UI, and mobile development.
# Meta-Preference Utilitarianism

There has been a long standing debate between utilitarians about what should be maximized. Most fall on either the side of ‘average utility’ or ‘total utility’. A select group even chooses ‘minimum utility’ or other lesser known methods.

Previously I have tried to solve this by charting different moral-theories on different axis and by prioritizing those actions that achieve success in most moral theories (see Q-balance utilitarianism).

I have come to the conclusion that this is just a bandaid on a more fundamental problem. Whether we should choose total, average or even median utility isn’t something we could objectively decide. So I suggest that we go up one level, and maximize what most people want to maximize.

Let’s say we were able to gauge everyone’s (underlying) preferences about how much they like certain methods of maximizing by holding a so called [utilitarian vote](https://en.wikipedia.org/wiki/Score_voting).

A utilitarian method that would violate your preferences completely would get a score of 0, one that would encapsulate your preferences perfectly would get a score of 1 and one that you like to a certain extent but not completely gets e.g 0,732.

If there is such a thing as ‘meta-preference ambivalence’ we could gauge that too: “People who do not have any meta-preferences in their utility-function get a score of 0, people for whom the entire purpose in life is the promotion of average utilitarianism will get a score of 1 etc.

Just multiply the ambivalence with the meta-preference and then add all the scores of the individual methods together (add all the scores of the preferences for “median utility” together, add all the scores for “total utility” together etc) and compare.

Now one way of maximizing comes out on top, but should we pursue it absolutely or proportionally? If “total utilitarianism” wins with 70% and “average utilitarianism” loses with 30% of the vote, should we act as “total utilitarians” 100% of the time or 70% of the time? (with average utilitarianism 30% of the time at random intervals)

Well, we could solve that with a vote too: “Would you prefer pursuing the winning method 100% of the time or would you prefer pursuing it proportionally to it’s victory?”

And any other problem we’ll face after this will be decided by comparing people’s preferences about those problems as well. (This might, depending on how you interpret it, also solve the negative-positive utilitarianism debate)

This might be encroaching on contractualism, but I think this is a very elegant way to solve some of the meta-level disagreements in utilitarian philosophy.
# Open & Welcome Thread - February 2020

If it’s worth saying, but not worth its own post, here's a place to put it. (You can also make a [shortform post](http://www.lesslong.com/))

And, if you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).