Harley-ml/lesswrong huggingface dataset:
# Intertheoretic utility comparison: examples

A previous [post](https://agentfoundations.org/item?id=1418) introduced the theory of intertheoretic utility comparison. This post will give examples of how to do that comparison, by normalising individual utility functions.

## The methods

All methods presented here obey the axioms of **Relevant data**, **Continuity**, **Individual normalisation**, and **Symmetry**. Later, we'll see which ones follow **Utility reflection**, **Cloning indifference**, **Weak irrelevance**, and **Strong irrelevance**.

# Max, min, mean

The maximum of a utility function $u$ is $\max_{s\in \mathbb{S}} u(s)$, while the minimum is $\min_{s\in \mathbb{S}} u(s)$. The mean of $u$ $\sum_{s\in \mathbb{S}} u(s)/||\mathbb{S}||$.

* The max-min normalisation of $[u]$ is the $u\in [u]$ such that the maximum of $u$ is $1$ and the minimum is $0$.

* The max-mean normalisation of $[u]$ is the $u\in [u]$ such that the maximum of $u$ is $1$ and the mean is $0$.

The max-mean normalisation has an interesting feature: it's precisely the amount of utility that an agent completely ignorant of its own utility, would pay to discover that utility (as a otherwise the agent would employ a random, 'mean', strategy).

For completeness, there is also:

* The mean-min normalisation of $[u]$ is the $u\in [u]$ such that the mean of $u$ is $1$ and the minimum is $0$.

# Controlling the spread

The last two methods find ways of controlling the spread of possible utilities. For any utility $u$, define the mean difference: $\sum_{s,s'\in\mathbb{S}} |u(s)-u(s')|$. And define the variance: $\sum_{s\in\mathbb{S}} (u(s)-\mu)^2$, where $\mu$ is the mean defined previously.

These lead naturally to:

* The mean difference normalisation of $[u]$ is the $u\in [u]$ such that $u$ has a mean difference of $1$.

* The variance normalisation of $[u]$ is the $u\in [u]$ such that $u$ has a variance of $1$.

## Properties

The different normalisation methods obey the following axioms:

| Property             | Max-min | Max-mean | Mean-min | Mean difference | Variance |
|----------------------|---------|----------|----------|-----------------|----------|
| Utility reflection   | YES     | NO       | NO       | YES             | YES      |
| Cloning indifference | YES     | NO       | NO       | NO              | NO       |
| Weak Irrelevance     | YES     | YES      | YES      | NO              | YES      |
| Strong Irrelevance   | YES     | YES      | YES      | NO              | NO       |


As can be seen, max-min normalisation, despite its crudeness, is the only one that obeys all the properties. If we have a measure on $\mathbb{S}$, then ignoring the cloning axiom becomes more reasonable. Strong irrelevance can in fact be seen as an anti-variance; it's because of its second order aspect that it fails this.
# The National Security Commission on Artificial Intelligence Wants You (to submit essays and articles on the future of government AI policy)

The National Security Commission on Artificial Intelligence, a body created by Congress to study the impact of advances in AI on US national security has put out a [call for essays](https://warontherocks.com/2019/07/in-search-of-ideas-the-national-security-commission-on-artificial-intelligence-wants-you/) which analyze how advances in artificial intelligence could affect US government and security policy. There are 5 specific prompts, which I summarize as follows:

1. How will AI affect the nature of war, and more generally interstate competition at or below the level of armed conflict?
2. What kind of military and non-military AI capabilities should the US government invest in? What AI-related skills will be necessary for the national-security workforce in the future?
3. What kind of infrastructure, institutions and organizational structures will be best suited for ensuring AI development?
    1. What kinds of AI research should the US national security community engage in?
    2. Will this research require the creation of new institutions?
    3. What other kinds of infrastructure are necessary to create a sustainable advantage in artificial intelligence, and what are the ethical concerns with attempting to create a sustainable advantage?
    4. How will government acquisition processes need to change?
    5. What kinds of data are necessary for developing AI applications and tools? What are the ethical and security concerns related to collecting, analyzing and storing this data?
4. What should the US do to influence global norms around artificial intelligence? Given that there are many nations pursuing AI, what should the US do to influence adversaries' AI development?
5. How should the government interact with the private sector? How can the private sector educate the government about the capabilities and risks of AI? What can the government and the private sector do that AI systems used for national security are trusted, by the public, strategic decision-makers, and allies?

I'm heartened to see the government taking the risks of AI more seriously, and I think submitting an essay here is a relatively low-effort way that someone, as a member of the public, can get their opinions regarding the risks and opportunities around AI development in front of a body specifically tasked with creating recommendations for future AI development and regulation.
# Thoughts on the 5-10 Problem

5 dollars is better than 10 dollars
===================================

The [5-10 Problem](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/zcPLNNw4wgBX5k8kQ) is a strange issue in which an agent reasoning about itself makes an obviously wrong choice.

Our agent faces a truly harrowing choice: it must decide between taking $5 (utility 5) or $10 (utility 10).

How will our agent solve this dilemna? First, it will spend some time looking for a proof that taking $5 is better than taking $10. If it can find one, it will take the $5. Otherwise, it will take the $10.

Fair enough, you think. Surely the agent will concede that it can't prove taking $5 is better than taking $10. Then, it will do the sensible thing and take the $10, right?

Wrong.

Our agent finds the following the following proof that taking $5 is better:

 F→(U=5)F→(¬F→(U=0))□((F→(U=5))∧(¬F→(U=0)))→F□((F→(U=5))∧(¬F→(U=0)))→((F→(U=5))∧(¬F→(U=0)))(F→(U=5))∧(¬F→(U=0))F

Let's go over the proof.

  

Line 1: Taking $5 gives you $5.

Line 2: If F is true, then ~F->x is true for any x.

Line 3: If you find a proof that taking $5 gives you $5 and take $10 gives you $0, you'd take the $5.

Line 4: Combine the three previous lines

Line 5: [Löb's Theorem](https://www.lesswrong.com/posts/ALCnqX6Xx8bpFMZq3/the-cartoon-guide-to-loeb-s-theorem)

Line 6: Knowing that taking $5 gives you $5 and taking $10 gives you $0, you happily take the $5.

Simplified Example
==================

To understand what went wrong, we'll consider a simpler example. Suppose you have a choice between drinking coffee (utility 1) and killing yourself (utility -100).

You decide to use the following algorithm: "if I can prove that I will kill myself, then I'll kill myself. Otherwise, I'll drink coffee".

And because a proof that you'll kill yourself, implies that you'll kill yourself, by Lob's Theorem, you will kill yourself.

Here, it is easier to see what went wrong-proving that you'll kill yourself is not a good reason to kill yourself.

This is hidden in the original 5-10 problem. The first conditional is equivalent to "if I can prove I will take $5, then I'll take $5".

Hopefully, it's now more clear what went wrong. How can we fix it?

Solution?
=========

I once saw a comment suggesting that the agent instead reason about how a similar agent would act (I can't find it anymore, sorry). However, this notion was not formalized. I propose the following formalization:

We construct an agent  A. Each time A makes a decision, it increments an internal counter n, giving each decision a unique identity. A uses the following procedure to make decisions: for each action a, it considers the agent Aa,n. Aa,n is a copy of A (from when it was created), except that if Aa,n would make a decision with id n, it instead immediately takes action a. Then, if A can prove any of these agents has the maximum expected utility, it chooses the action corresponding to that agent.
# When does adding more people reliably make a system better?

Prediction markets have a remarkable property. They reward correct contrarianism. They incentivise people to disagree with the majority consensus, and be right. If you add more traders to a market, in expectation they price will be more accurate.

More traders means both more fish and more sharks.

(The movie "The Big Short" might be a very sad portrait of the global financial system. But it's still the case that a system in a bad equilibrium with deeply immoral consequences rewarded the outcasts who pointed out those consequences with _billions of dollars._ Even though _socially, no one bothered listening to them,_ including the US Government who ignored requests by one of the fund managers to share his expertise about the events _after_ the crash.)

Lots of things we care about don't have this property.

*   Many social communities decline as more members join, and have to spend huge amounts of effort building institutions and rituals to prevent this.
*   [Many companies have their culture decline as they hire more,](https://twitter.com/patrickc/status/1068236758616174592?lang=en) and have to spend an incredible amount of resources simply to p_revent_ this (which is far from getting b_etter_ as more people join). (E.g. big tech companies can probably have >=5 candidates spend >=10 hours in interviews for a a _single_ position. And that's not counting the probably >=50 candidates for that position spending >=1h.)
*   Online forums usually decline with growing user numbers (this happened to Reddit, HackerNews, as well as LessWrong 1.0).

In prediction markets the vetting process is really cheap. You might have to do some KYC, but mostly new people is great. This seems like a really imporant property for a system to have, and something we could learn from to build other such systems.

What other systems have this property?
# [Link] Intro to causal inference by Michael Nielsen (2012)

This is a link post for Michael Nielsen's "[If correlation doesn’t imply causation, then what does?](http://www.michaelnielsen.org/ddi/if-correlation-doesnt-imply-causation-then-what-does/)" (2012).

I want to highlight the post for a few reasons:

(1) it is a well-written introduction by an experienced science communicator — Michael is an author of the most famous book on quantum computing;

(2) causal inference is an essential tool for understanding the world;

(3) two recent AI safety papers use causal influence diagrams to (a) understand agent incentives \[[arXiv](https://arxiv.org/abs/1902.09980), [Medium](https://medium.com/@deepmindsafetyresearch/understanding-agent-incentives-with-causal-influence-diagrams-7262c2512486)\] and (b) to provide a new perspective on some problems in AGI safety \[[arXiv](https://arxiv.org/abs/1906.08663)\].
# What questions about the future would influence people’s actions today if they were informed by a prediction market? 

I'm looking for questions that would be useful to many individuals and would influence their decisions today (not questions for those working on shaping the long term future of Earth originating life).

I would also like if you specified how those questions might influence people's actions.

There's a couple of questions I would really like to have our current best collective prediction, and I really feel our civilisation is inadequate at providing those.

Edit: side question: where could I post those questions? long term (>5 years) bets need to invested on the stock market, otherwise there's no financial incentives in betting
# APL AI Safety Conference

The Johns Hopkins University Applied Physics Laboratory (APL) in Laurel, MD will be hosting a full-day conference on AI safety, on August 29. The speaker line-up has not been finalized yet, but current confirmed speakers include:

*   Max Tegmark
*   Lt. Gen. John “Jack” Shanahan, Director of the Joint AI Center of the U.S. Department of Defense
*   Bob Work, Former Deputy Secretary of Defense, currently serving as Co-Chair of the National Security Commission on AI

I will try to update this space when the speaker line-up is announced. Space is limited and advanced registration is required, so register now if you're interested in attending.

[https://www.eventbrite.com/e/the-future-of-humans-and-machines-assuring-artificial-intelligence-tickets-64320947686](https://www.eventbrite.com/e/the-future-of-humans-and-machines-assuring-artificial-intelligence-tickets-64320947686)
# "Shortform" vs "Scratchpad" or other names

Within the next day or two, we'll be launching some features that give the "shortform" concept real infrastructural support. Shortform comments will appear on the All Posts page, on the new [/shortform page](https://www.lesswrong.com/shortform), and it'll be easier to automatically generate a [post to write shortform](https://www.lesswrong.com/posts/jL7uDE5oH4HddYq4u) content.

The only potential issue is that, as "shortform" comments have gained traction... they... often aren't especially short.

I'd kinda like to finalize the name before it goes live.

Currently the team is looking at "Scratchpad" as a name, intending to convey "this is a place you write early stage, off the cuff ideas without stressing too much about whether it should be a post." Sometimes this turns out to actually be short, [sometimes not](https://www.lesswrong.com/posts/EQJfdqSaMcJyR5k73/habryka-s-shortform-feed#xQJYMNujxB3EqephZ). The main counterpoint right now is that scratchpad kinda implies a level of "ephemerality" or something that isn't necessarily true either.

I'm particularly interested in feedback from people who've either already made a shortform feed (i.e. mr-hire, hazard, DanielFilan and others) if they have any opinions about this particular [bikeshed](https://en.wiktionary.org/wiki/bikeshedding).
# The Real Rules Have No Exceptions

*(This is a [comment](https://www.lesswrong.com/posts/CKoyBW5T7XaNSkawp/making-exceptions-to-general-rules#PKPnY3AN7HR6cpXcR) that has been turned into a post.)*

From Chris_Leong’s post, “[Making Exceptions to General Rules](https://www.lesswrong.com/posts/CKoyBW5T7XaNSkawp/making-exceptions-to-general-rules#comments)”:

> Suppose you make a general rule, ie. “I won’t eat any cookies”. Then you encounter a situation that legitimately feels exceptional , “These are generally considered the best cookies in the entire state”. This tends to make people torn between two threads of reasoning:
> 
> 1) Clearly the optimal strategy is to make an exception this one time and then follow the rule the rest of the time.
> 
> 2) If you break the rule this one time, then you risk dismantling the rule and ending up not following it at all.
> 
> How can we resolve this? …

This is my answer:

*Consider even a single exception to totally undermine any rule. Consequently, only follow rules with no exceptions.[^1]. When you do encounter a legitimate exception to a heretofore-exceptionless rule, immediately discard the rule and replace it with a new rule—one which accounts for situations like this one, which, to the old rule, had to be exceptions.*

This, of course, requires a meta-rule (or, if you like, a meta-habit):

*Prefer simplicity in your rules. Be vigilant that your rules do not grow too complex; make sure you are not relaxing the legitimacy criteria of your exceptions. Periodically audit your rules, inspecting them for complexity; try to formulate simpler versions of complex rules.*

So, when you encounter an exception, you *neither* break the rule once but keep following it thereafter, *nor* break it once and risk breaking it again. If this is really an exception, then that rule is immediately and automatically nullified, because good rules ought not have exceptions. Time for a new rule.

And if you’re not prepared to discard the rule and formulate a new one, well, then the exception must not be all that compelling; in which case, of course, keep following the existing rule, now and henceforth.

But why do I say that good rules ought not have exceptions? Because **rules already don’t have exceptions**.

Exceptions are a fiction. They’re a way for us to avoid admitting (sometimes to ourselves, sometimes to others) that the rule as stated, *together with the criteria for deciding whether something is a “legitimate” exception*, **is the actual rule**.

The approach I describe above merely consists of making this fact explicit.

[^1]: By which I mean “only follow rules to which no legitimate exception will ever be encountered”, *not* “continue following a rule even if you encounter what seems like a legitimate exception”.
# How often are new ideas discovered in old papers?

Suppose someone wrote a paper about X two decades ago. A modern reader realizes the X paper sheds light on an unrelated idea Y. Do we have any information on how often this happens? How often is this just "I figured out Y for a different reason, and while doing my lit review I realized that the X paper is also relevant for Y"?
# Can you summarize highlights from Vernon's Creativity?

Gwern had responded to [Elizabeth's question about creativity](https://www.lesswrong.com/posts/smjMNYeS2myBpqYGn/how-can-we-measure-creativity) with the following:

> A good anthology to read is _[Creativity,](https://www.gwern.net/docs/psychology/writing/1970-vernon-creativity.pdf)_ ed Vernon 1970 - it's old but it shows you what people were thinking back when Torrance was trying to come up with creativity tests, and the many psychometric criticisms back then which I'm not sure have been convincingly resolved.

I'm not sure whether Elizabeth has already read it, but I'd be interested in reading the highlights from that if anyone was up for distilling it down into something more manageable.
# Results of LW Technical Background Survey

[See results here](https://docs.google.com/forms/d/1os_miirkd_iiGwRBLG5bJhHBgf0i9bcKg5-UgFIZkPo/viewanalytics).

The main goal of [the survey](https://www.lesswrong.com/posts/dohwEhsNofor5hFtd/how-much-background-technical-knowledge-do-lw-readers-have) was to provide info for authors about their target audience, so here's a high-level overview toward that end:

*   The average respondent is some kind of professional programmer, with an undergrad degree (or equivalent) in CS.
*   Most people have seen at least some economics and probability, but not at the level of a undergrad degree.
*   Almost everyone knows calculus, but linear algebra or differential equations will likely be lost on at least ~25% of respondents.
*   There are substantial zero-knowledge and high-knowledge counts for most areas.
*   About half of respondents had read the sequences in their entirety.

[Here are charts](https://docs.google.com/forms/d/1os_miirkd_iiGwRBLG5bJhHBgf0i9bcKg5-UgFIZkPo/viewanalytics) of the responses to each question. I strongly recommend looking at them directly rather than just taking my summary at face value. As always, remember this is an opt-in survey without any sort of verification of responses, so take everything with a grain of salt.

One interesting note: we had a handful of respondents declaring very high skill levels (Nobel-level economists, Turing-level computer scientists, or primary developers of popular software). I'd personally be interested to hear what exactly those people work on, especially if they're willing to occasionally field questions on their area of expertise. All y'all should leave a comment or something.

Actually, I'm curious what everyone works on, especially specialties for all the researchers. Feel free to leave a quick comment, especially if you're able and willing to occasionally field questions in your area of expertise.
# Compilers/PLs book recommendation?

I'm interested in:

*   Data structures for representing programs
*   Algorithms for translating between those representations
*   Analysis & manipulations of each representation; questions which are natural to ask/answer in each

Most books I can find on compilers/PLs tend to spend most of their time on the text representation (and algorithms for translating programs out of text, i.e. parsing) and the machine-code representation (and algorithms for translating programs into machine code). For purposes of this question, I'm not particularly interested in either of these representations - they're not very natural data structures for representing programs, and we mostly use them because we have to.

I'm interested mainly in high-level "intermediate" representations. In terms of applications, I'm more interested in reasoning about about the code (e.g. control flow analysis, type checking) and manipulating the code (e.g. automated parallelization, high-level machine-independent optimization methods) rather than translating between any particular source/target format.

Questions:

*   Does anybody know of a good book (or other source) that focuses on high-level "intermediate" representations of programs?
*   Is there some other question I should be asking, e.g. a different term to search for?
*   On the meta-level, where else should I look/ask this question?
# Information empathy

_Epistemic status: Giving a name to a basic skill that some people exhibit and others don't, because I think that it's useful to keep track of who exhibits it and that giving it a name will help with this. See also: [Historian's Fallacy](https://en.wikipedia.org/wiki/Historian%27s_fallacy), [Theory of Mind](https://en.wikipedia.org/wiki/Theory_of_mind) (both Wikipedia links)._

**Edit:** Down below, I was convinced that this is **not a useful term** because it is nearly identical in meaning to Theory of Mind, and that the latter can be substituted in whenever you would use this term. This post might still be worth reading if you don't know what that is, but please read mentally substitute the official phrasing for my coinage.

* * *

The experiment I'm about to describe may be apocryphal (links, anyone?), but it illustrates my point nicely. (Edit: down below, Raemon has linked the [real experiment](https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test), which is slightly different but has exactly the same moral.)

In the experiment, a child is shown a box with a "trick" lid, such that when the lid is on the box it appears that there is a cookie inside, but when the lid is removed you can see that it is in fact empty. First, with the lid on the box, the child is asked what is inside. "A cookie," they answer. Then the lid is removed, and again the child is asked what is inside. "Nothing," they say. Finally, the lid is replaced, and the experimenter tells the child that they are going to show the box to another child, who has not yet seen the box with the lid off.

_"What will the other child think is in the box?"_ the experimenter asks.

At this point, things can go one of two ways. Children beyond a certain developmental stage will say "she'll say there's a cookie", while children who have not yet reached this stage will say "she'll say it's empty".

I call this skill **Information Empathy**, and from what I can tell, the median age at which children develop it is 35.

Jokes aside, I feel like there have been actual occasions where somebody has been angry at me after I did _X_, where the primary reason for their anger seems to be neither that _X_ happened to them nor that my doing _X_ was careless negligent but rather that I _did X knowing full well what the consequences were_... even when they could see that that was not true.

Maybe this term will be useful for some of you.
# Does improved introspection cause rationalisation to become less noticeable?

I've recently updated that [noticing](https://www.lesswrong.com/posts/o7W3oB8SR679HGeAq/what-it-s-like-to-notice-things) is a key rationality skill -- not just noticing confusion, but noticing your cognition more generally. This allows you to figure out at a very granular level why you're not reaching your goals, and then intervene to change those reasons.

For example:

> At one point I found myself procrastinating on ordering the catering for an event. Noticing the disconnect between my high-level goals ("make a good event") and my concrete actions ("spend time on FB"), triggered me to try to notice what was up in my mind (this is a particular trigger-response pattern I've trained myself to use). I found that I didn't want to make the call since last time I called them, they couldn't hear what I was saying and were kind of rude about it. I didn't want my phone to be bad or my accent to be inaudible, an so I didn't want to call them again. I then proceeded to borrow a friend's phone, and called them without problem.

Another example, this time with a concrete cognitive rather than practical intervention:

> I noticed myself being unhappier than I wanted to be. So when the unhappiness clashed with the higher-level desire for happiness, it triggered a noticing process, and I realised my mind was running an algorithm like: "notice happy thought --> remember Hamming problem or that timelines might be short --> feel bad". This sounds ridicolously unhelpful when written out, but is in fact what was going on. So I started training myself to hold on to the happiness in the first part of the chain without automatically falling into the second.

  

Here's a worry with this: if part of my congition is consciously accessible and interpretable, and part of it is not, will extensive noticing-and-intervening cause motivated cognition to become less noticable?

It will by selection effects, since the more noticeable parts I'll change. But this feels more definitionally true than actually worrying.

It also might by negative reinforcement, if my mind learns that when subagents make their desires known they'll tend to be overruled/modified. (To prevent this, and as a safer policy in my current epistemic state, I make sure to sometimes deliberately not intervene on things I've noticed.) But this shouldn't be the case if I genuinely listen to subagents and take their preferences into account; as well as if the subagent theory doesn't fit (which seems more plausible in the second example above).

  

Is there some other reason to believe that improved ability to notice your cognition will cause rationalisation, motivated cognition, thought patterns highly valued by certain subagents, etc. to become _less_ noticeable?
# Gathering thoughts on Distillation

I think one of the biggest outstanding issues facing LessWrong is good infrastructure for distilling lengthy conversations.

Wei_Dai notes in the [comments of Forum Participation as Research Strategy](https://www.lesswrong.com/posts/rBkZvbGDQZhEymReM/forum-participation-as-a-research-strategy#EAC8PDgcDz2F6Rtmj):

> There's nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren't doing that, or are doing that less than they should, that could potentially be solved as a problem that's separate from "should more people be doing FP or traditional research?"

> Also, it's not clear to me that traditional research produces more clear distillations of how disagreements get resolved. It seems like most such discussions don't make for publishable papers and therefore most disagreements between "traditional researchers" just don't get resolved in a way that leaves a public record (or at all).

The LessWrong team has discussed this on and off for about 9 months. While we haven't done any development work on it yet (mostly working on Open Questions instead), habryka and I at least had some sense that this might be one of the most important things LessWrong is missing.

I have some vague thoughts on what would be needed here, but am interested in others opinions.

The rough thing that seems missing to me is a pipeline that goes something like:

1\. Somebody notices that there was either a post that was way longer than it needed to be (or a bit confusing, or something). _Or_ a meandering comment conversation that was could use to be distilled down into a post. They flag that collection of posts and comments into a databse object that says "hey, this cluster of posts and/or comments could use distillation"

2\. Someone (sometimes that same person, sometimes a totally different person), writes up a distillation of that cluster of posts and/or comments

3\. It might be that different people write up _different_ distillations of the same content, because they had different takeaways. Or maybe two people both agreed that a post was long, but they both found different central metaphors useful.

4\. The people who wrote the original post or comments can mark distillations as "good representation of my views" or not.

In theory there's nothing preventing this from happening with the current site technology. But, a year ago, there was nothing stopping people from making question posts (and getting answers). And yet introducing the question feature made it much more salient that "ask questions" is a thing you're encouraged to do, and then people did it more.

I think having some kind of official distillation pipeline could be useful in a similar way.

Interested in people's thoughts on this.
# Do you use twitter for intellectual engagement? Do you like it?

By happenstance I never started using social media much, and when I stumbled onto the Deep Work mindset I began intentionally avoiding it.

One thing I'd like in my life is an intellectual community, where I can routinely go to find people's thoughts that I'm interested, routinely produce thoughts people are interested in, and engage and be engaged with on those ideas. A few things I've seen recently have slightly updated me that twitter might be useful for this.

I've already seen [Zvi](https://thezvi.wordpress.com/2017/04/22/against-facebook/) and [others](http://www.thelastrationalist.com/facebook-the-rodents-and-the-common-knowledge-machine.html) talk against facebook, but I haven't heard many people talk about twitter.

What has your experience been with using twitter to find, engage with, and share interesting ideas?
# [Site Update] Weekly/Monthly/Yearly on All Posts

Last week, our friends over at the EA Forum coded a new feature for the /allPosts page – in addition to the daily view, you can now view posts by weekly, monthly and yearly. (Thanks JP!)

This is most exciting when you also set the sorting to "top karma", making it an easy way to catch up on the most important posts you missed.

For convenience, sorted by top-karma, here are links to:

*   [The Weekly View](https://www.lesswrong.com/allPosts?filter=all&sortedBy=top&timeframe=weekly)
*   [The Monthly View](https://www.lesswrong.com/allPosts?filter=all&sortedBy=top&timeframe=monthly)
*   [The Yearly View](https://www.lesswrong.com/allPosts?filter=all&sortedBy=top&timeframe=yearly)

Have fun perusing posts you haven't read yet, or may have forgotten about, and perhaps catching up on newer comments on some of the top discussions. :)
# Open problems in human rationality: guesses

A couple months back Raemon wrote this [excellent question](https://www.lesswrong.com/posts/gX8fcAwk3HGkFyJgk/what-are-the-open-problems-in-human-rationality) to which Scott Alexander shared his [ongoing list](https://www.lesswrong.com/posts/gX8fcAwk3HGkFyJgk/what-are-the-open-problems-in-human-rationality#o3WG8eMzuQ5pKEiyD). I think it would be great to have people try to give their current guesses for a lot of these. My guesses in a comment below. My intuition is that value is created in four ways from this:

1\. Discovery of things you didn't realize you believed in the process of writing the answer.

2\. Generation of cruxes if people give you feedback/alternatives for answers.

3\. Realization that your guess isn't even wrong, but fundamentally wasn't built from building blocks that can, in principle, be rearranged to form a correct answer.

4\. Help in coordination as people get a sense of what others believe about navigating this space. Seeing cognitive diversity on fundamental questions has helped me in this area.
# Very different, very adequate outcomes

Let $U_p$ be the utility function that - somehow - expresses your preferences[^pref]. Let $U_h$ be the utility function expresses your hedonistic pleasure.

[^pref]: This being a somewhat naive form of preference utilitarianism, along the lines of "if the human choose it, then its ok". In particular, you can end up in equilibriums where you are miserable, but unwilling to choose not to be (see for example, some forms of depression).

Now imagine an AI is programmed to maximise $U(q)=qU_p + (1-q)U_h$. If we vary $q$ in the range of $5\%$ to $95\%$, then we will get very different outcomes. At $5\%$, we will generally be hedonically satisfied, and our preferences will be followed if they don't cause us to be unhappy. At $95\%$, we will accomplish any preference that doesn't cause us huge amounts of misery.

It's clear that, extrapolated over the whole future of the universe, these could lead to very different outcomes[^trade]. But - and this is the crucial point - none of these outcomes are really that bad. None of them are the disasters that could happen if we picked a random utility $U$. So, for all their differences, they reside in the same nebulous category of "yeah, that's an ok outcome." Of course, we would have preferences as to where $q$ lies exactly, but few of us would risk the survival of the universe to yank $q$ around within that range.

[^trade]: This fails to be true if preference and hedonism can be maximised independently; eg if we could take an effective happy pill and still follow all our preferences. I'll focus on the situation where there are true tradeoffs between preference and hedonism.

What happens when we push $q$ towards the edges? Pushing $q$ towards $0$ seems a clear disaster: we're happy, but none of our preferences are respected; we basically don't matter as agents interacting with the universe any more. Pushing $q$ towards $1$ *might* be a disaster: we could end up always miserable, even as our preferences are fully followed. The only thing protecting us from that fate is the fact that our preferences include hedonistic pleasure; but this might not be the case in all circumstances. So moving $q$ to the edges is risky in the way that moving around in the middle is not.

In my [research agenda](https://www.lesswrong.com/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into), I talk about adequate outcomes, given a choice of parameters, or acceptable approximations. I mean these terms in the sense of the example above: the outcomes may vary tremendously from one another, given the parameters or the approximation. Nevertheless, all the outcomes avoid disasters and are clearly better than maximising a random utility function.
# Open & Welcome Thread - August 2019

*   If it’s worth saying, but not worth its own post, here's a place to put it.

*   You can also make a [shortform post](http://www.lesslong.com).

*   And, if you are new to LessWrong, here's the place to introduce yourself.

*   Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Proposed algorithm to fight anchoring bias

[Anchoring](https://en.wikipedia.org/wiki/Anchoring) is a classic cognitive bias which has been [discussed on Less Wrong before](https://www.lesswrong.com/posts/bMkCEZoBNhgRBtzoj/anchoring-and-adjustment).  Anchoring seems very [difficult to avoid](https://en.wikipedia.org/wiki/Anchoring#Difficulty_of_avoiding).  Experiments have found that warning subjects about anchoring, or giving them cash incentives, doesn't solve the problem.

Here's an algorithm to fight anchoring that I would like to see a researcher test, based on [binary search](https://en.wikipedia.org/wiki/Binary_search_algorithm):

1. Tell subjects to think of a number which is clearly too high for the quantity they want to estimate (an upper bound).
2. Tell subjects to think of a number which is clearly too low (a lower bound).
3. Tell subjects to find the midpoint of the upper bound and the lower bound and figure out whether it's too high or too low.
4. The midpoint has now been judged as an upper/lower bound.  Combined with the original lower/upper bound, we have a new, narrower range to explore.  If this range is narrow enough, report its midpoint; otherwise go to step 3.

You could have two experimental conditions: one condition where subjects think of a number which is clearly too high first (the steps are in the order above), and another condition where subjects think of a number which is clearly too low first (steps 1 & 2 are swapped).  If estimates from the two conditions are similar, the technique is successful.
# Zeno walks into a bar

Zeno walks into a bar.  
  
"I have a problem," he said.  
  
"What is it?" said the bartender.  
  
"Well, it has to do with the movement of physical bodies," said Zeno.  
  
"Talk to my friend Max," said the bartender. He gestured toward a German man wearing round spectacles.  
  
"Sir," said Zeno, "I wonder if you could help me with a problem."  
  
"What's the problem?" said Max.  
  
"Suppose I shoot an arrow from point  A to point B," said Zeno. "Before it reaches point B it must first reach a point C1 midway between points A and B."  
  
"Naturally," said Max.  
  
"And before the arrow reaches point C1 it must reach a point C2 midway between points C1 and A," continued Zeno.  
  
"I see," said Max.  
  
"And before the arrow reaches point C2 it must reach a point C3 midway between points C2 and A," continued Zeno.  
  
"Wait a minute," said Max. "How far apart are points A and B?"  
  
"10 meters," said Zeno.  
  
"Then yes," said Max. "I understand your situation."  
  
"And before the arrow reaches point C3 it must reach a point C4 midway between points C3 and A," continued Zeno. "Do you see the impasse?"  
  
"Nope," said Max, "I think we're getting somewhere. How long is the arrow?"  
  
"One meter," said Zeno.  
  
"The distance between points C3 and C4 is five eighths of a meter," said Max. "A one-meter-long arrow can be at point C3 and C4 at the same time."  
  
"Let's consider the tip of the arrow then," said Zeno. "Before the tip of the arrow reaches point C3 it must reach a point C4 midway between points C3 and A."  
  
They talked deep into the night.  
  
"And before the high-energy particle reaches point C118 it must reach a point C119 midway between points C118 and A," continued Zeno.  
  
"Hold on," said Max. "How far apart are points A and C119?"  
  
"1.5×10−35 meters" said Zeno.  
  
"That's shorter than 1.6×10−35 meters," said Max. "The uncertainty in the position of a particle must always exceed 1.6×10−35 meters, because of space-time equivalence and the quantum-mechanical velocity operator's non-commutation with position. Even theoretically, the wave function of a particle can't ever occupy a space smaller than 1.6×10−35 meters."  
  
"Thanks," said Zeno.  
  
"By the way," said Max, "What brought you to this question in the first place?"  
  
"I wanted to know how to define the momentum of a particle at an instantaneous moment of time," said Zeno.  
  
"You could have just asked," said Max. "The probability distribution of a particle's momentum is determined by the instantaneous phase and magnitude of its wave."
# Do you do weekly or daily reviews? What are they like?

* What system(s) do you use to keep yourself organized and working toward your goals? I'm interested in technologies, but more interested in what ontology you use to organize your tasks, events, goals, and plans.
* Did you build your system slowly over time, or adopt it all at once?
* What feels most important about your system to you?
# How to navigate through contradictory (health/fitness) advice?

I will start with a brief story, but the question can be generalized.

Last year, I decided to do something for my body. I joined and regularly went to K. Training (abbreviated name), a large gym chain in german-speaking countries. The claimed philosophy is different from many gyms: there is no music, no proteine shakes to buy, mostly old people around, and insistence that it is about strength, not show-off, and that strength is what keeps your spine together etc. They have no cardio bikes, no barbell, only machines, and the high-intensity approach is that at each machine you do one continuous exercise for two minutes. If you reach the two minutes, increase the weight next time. It all seems very serious, there is an orthopedist you talk to when you become a member. It all has been in existence for some decades. The founder writes books, of course mentioning that his approach is the only one that works against pain, and that he is not heard by the mainstream. While at the same time they have contracts with many orthopedists and this is part of the marketing.

Now, a back problem. I have seen several orthopedists in my life, but the one I talked to this year (after two GPs, both clueless) is the first who seems competent and also listens. His comment about K. Training: it's ok, but sometimes hard to leave the contract. You could just as well try Yoga or Pilates. Anyways, he gives me a prescription for physical therapy. 

Talking about this and that, the therapist speaks out against K Training, because no warming up / cardio (something the founder explicitly defends in his books), and Yoga/Pilates/etc is better anyways.

Then I googled again. Seemingly, gym experts all have their own approach. Some agree to the high-intensity two minutes thing, others disagree.

Then there is also Mr. L.-B., an anti-pain guru with a somewhat different approach I dont really understand, again against the "mainstream" but also against K. Training. And from a lecture of his that I watched on youtube, he seems like a snake-oil seller; but then, he (of course) has many fans.

Now I could just randomize what to do; or try to really read about approaches, but ALL of them seem plausible, if you listen to them. The investment necessary for actual judgement would be studying medicine.

So long story, short question: how do you actually handle such cases of pratically relevant epistemic learned helplessness?
# What are the best resources for examining the evidence for anthropogenic climate change?

A while back I was researching the evidence for evolution. It's not that I didn't initially believe in evolution or understand natural selection, but it's just that I didn't really understand the full breadth of the evidence and predictions that the theory makes. Before, I had a tendency to simply assert that "the evidence is overwhelming" in discussions without really going into detail.

When I began researching evidence, I had a few choices. I could just read basic surface arguments that I found on the internet, such as [this article](https://www.khanacademy.org/science/biology/her/evolution-and-natural-selection/a/lines-of-evidence-for-evolution) from Khan academy or the [Wikipedia page](https://en.wikipedia.org/wiki/Evidence_of_common_descent). While these resources are valuable, they aren't very comprehensive, and don't appear like they'd convince a hard-nosed skeptic. There are popular books, such as Jerry A. Coyne's [Why Evolution Is True](https://www.amazon.com/Why-Evolution-True-Jerry-Coyne/dp/0143116649) and Richard Dawkins' [The Greatest Show on Earth](https://www.amazon.com/Greatest-Show-Earth-Evidence-Evolution/dp/1416594795). The last two sources left me feeling like I still wasn't getting the full story, since they assumed a beginner background in philosophy and science, and weren't as nuanced as I wanted them to be (although I did not read both of them cover to cover).

Eventually I stumbled across [29+ Evidences for Macroevolution: _The Scientific Case for Common Descent_](http://www.talkorigins.org/faqs/comdesc/) by Douglas Theobald, which exceeded my expectations, and satisfied my desire to understand the evidence for common descent. While this last work does not assume the reader is a professional biologist, it also doesn't shy away from presenting specific technical evidences and the context they play in modern biology.

I wonder whether there is a similar publication which can satisfy my desire to understand anthropogentic climate change. My prior is that climate change is real, and primarily caused by human activity. I believe this because I generally side with the scientific consensus, and most intelligent people I know believe it. However, I am a little embarrassed from the fact that I couldn't really convincingly argue with a skeptic. I imagine a highly educated climate change skeptic like [Roy Spencer](https://en.wikipedia.org/wiki/Roy_Spencer_(scientist)) could argue circles around me, which is never a good sign.

In light of the previous discussion, what are the best resources for understanding the full breadth of evidence for anthropogenic climate change?
# Status 451 on Diagnosis: Russell Aphasia

Samuel the Fifth at _Status 451_ considers the implications of [Russell conjugation](https://www.edge.org/response-detail/27181), or [emotive conjugation](https://en.wikipedia.org/wiki/Emotive_conjugation) (example: she lied; you distorted the truth; I [honestly reported unconsciously biased beliefs](https://slatestarcodex.com/2019/07/16/against-lie-inflation/)). June 2019, ~1200 words.

Samuel emphasizes that this phenomenon isn't merely a matter of perceiving the outgroup in a negative light. Even if the different "forms" in a Russell-conjugation triplet are synonyms, they're not _exact_ synonyms: "lie" and "rationalize" _actually mean different things_. The inability to separate the _denotative_ content of language from an imputed [enactive](https://www.lesswrong.com/posts/i2bWqSFgyFxowTKWE/actors-and-scribes-words-and-deeds) "side-channel attack" about [who is to be blamed](https://www.lesswrong.com/posts/r2dTchodfqX4o5DYH/blame-games) is a _critical_ threat to our collective ability to construct shared maps that reflect the territory! Key quote:

> A culture that cannot tell the difference between "reporting" and "doxing" and merely considers it "doxing" when _they_ do it, is a culture that cannot accurately talk about behavior anymore. It's a culture where the words have been spun so much, that they have lost their objective meaning, and are now instead used more interchangeably to deliver more subjective messages. It's also a culture where not enough people feel the _need_ to define their words, and to actually figure out what counts as reporting, as gossiping, as doxing, because this way the words can be used strategically in the moment, without having to be fully consistent with next time.
# Why is the nitrogen cycle so under-emphasized compared to climate change

While listing various environmental issues of how [planetary boundaries](https://www.stockholmresilience.org/research/planetary-boundaries/planetary-boundaries/about-the-research/the-nine-planetary-boundaries.html) are exceeded the Stockholm Resilience Centre lists climate change in the "increasing risk category" while listing "Nitrogen and phosphorus flows to the biosphere and oceans" in the high risk category which suggests that they consider it a more grave environmental issue.

Do I misread them? Otherwise, are there views inline with other scientists in their field?

If that's a consensus view, view do we have so much more debate about climate change but none about nitrogen and phosphorus flows?