Harley-ml/lesswrong huggingface dataset:
# Bounded rationality abounds in models, not explicitly defined

Last night, I did not register a patent to cure all forms of cancer. Even though it’s probably possible to figure such a cure out, from basic physics and maybe a download of easily available biology research papers.

Can we then conclude that I don’t want cancer to be cured – or, alternatively, that I am pathologically modest and shy, and thus don’t want the money and fame that would accrue?

No. The correct and obvious answer is that I am boundedly rational. And though an unboundedly rational agent – and maybe a superintelligence – could figure out a cure for cancer from first principles, poor limited me certainly can’t.

Modelling bounded rationality is tricky, and it is often accomplished by artificially limiting the action set/action space. Many economic models use revealed preferences, and feature agents that are assumed to be fully rational, but who are restricted to choosing between a tiny set of possible goods or lotteries. They don’t have the options of developing new technologies, rousing the population to rebellion, going online and fishing around for functional substitutes, founding new political movements, begging, befriending people who already have the desired goods, setting up GoFundMe pages, and so on.

There’s nothing wrong with modelling bounded rationality via action set restriction, as long as we’re aware of what we’re doing. In particular, we can’t naively conclude that because a such a model fits with observation, that therefore humans actually are fully rational agents. In particular, though economists are right that humans are more rational than we might naively suppose, thinking of us as rational, or “mostly rational”, is a colossally erroneous way of thinking. In terms of achieving our goals, as compared with a rational agent, we are barely above agents acting randomly.

Another problem with using small action sets, is that it may lead us to think that an AI might be similarly restricted. That is unlikely to be the case; an intelligent robot walking around would certainly have access to actions that no human would, and possibly ones we couldn’t easily imagine.

Finally, though action set reduction can work well in toy models, it is wrong about the world and about humans. So as we make more and more sophisticated models, there will come a time when we have to discard it, and tackle head-on the difficult issue of defining bounded rationality properly. And it’s mainly for this last point I’m writing this post; we’ll never see the necessity of better ways of defining bounded rationality, unless we realise that modelling it via action set restriction is a) common, b) useful, and c) wrong.
# Introducing the Longevity Research Institute

I’ve just founded a nonprofit, the Longevity Research Institute — you can check it out [here.](https://thelri.org/)

The basic premise is: we know there are more than 50 compounds that have been reported to extend healthy lifespan in mammals, but most of these have never been tested independently, and in many cases the experimental methodology is poor.

In other words, there seems to be a _lot _of low-hanging fruit in aging.  There are many long-lived mutant strains of mice (and invertebrates), there are many candidate anti-aging drugs, but very few of these drugs have actually been tested rigorously.

Why?  It’s an incentives problem.  Lifespan studies for mice take 2-4 years, which don’t play well with the fast pace of publication that academics want; and the FDA doesn’t consider aging a disease, so testing lifespan isn’t on biotech companies’ critical path to getting a drug approved.  Mammalian lifespan studies are an underfunded area — which is where we come in.

We write grants to academic researchers and commission studies from contract research organizations.  Our first planned studies are on epitalon (a peptide derived from the pineal gland, which has been reported to extend life in mice, rats, and humans, but only in Russian studies) and C3 carboxyfullerene (yes, a modified buckyball, which prevents Parkinsonism in primate models and has been reported to extend life in mice).  I’m also working on a paper with [Vium](http://www.vium.com/) about some of their long-lived mice, and a quantitative network analysis of aging regulatory pathways that might turn up some drug targets.

We’re currently fundraising, so if this sounds interesting, please consider donating. The more studies that can be launched in parallel, the sooner we can get results.
# Gwern about centaurs: there is no chance that any useful man+machine combination will work together for more than 10 years, as humans soon will be only a liability

This is a quote from the discussion in Reddit (responding to another commenter):

> "I remember and I kind of subscribe to this idea of "man+machines as a future of work". I think Tyler Cowen implanted this idea into my head"

> Yes, Cowen was big on that too in stuff like Average is Over.

> Very irresponsible of them to try to foster complacency like that. It should have been beyond obvious that there was no reason chess engines wouldn't keep improving and that at some point very quickly, far from representing a new stable paradigm and a reason to not worry about technological unemployment, the 'centaur' would be a net liability. As far as I can tell, in chess, the centaur era lasted barely a decade, and would've been shorter still had anyone been seriously researching computer chess rather than disbanding research after Deep Blue or the centaur tournaments kept running instead of stopping a while ago. In Go, it lasted a year at best (if we assume that world champs like Lee Sedol could spot 'delusions' like made it lose a game to Lee Sedol and contribute at all, but then by the Ke Jie tournament with Master, between Master's performance and the various match settings, it looked like humans were way far behind and liabilities when paired with Master, and even if we doubt that, Zero then came out and superseded Master entirely). Not very comforting precedents... The idea was nice but it doesn't work.
# Sabine "Bee" Hossenfelder (and Robin Hanson) on How to fix Academia with Prediction Markets

"Don’t ask what science can do for you." is the clever title.

As I understand it, the prediction markets in science would not be easily influenced by the scientists and they would take into account a group of people currently completely neglected in scientific evaluations: those who left academia dissatisfied. Not a new approach in the industry, but certainly not something that has been tried in a research-oriented environment, as far as I know.
# The E-Coli Test for AI Alignment

Let’s say you have an idea in mind for how to align an AI with human values.

Go prep a slide with some e-coli, put it under a microscope, and zoom in until you can see four or five cells. Your mission: satisfy the values of those particular e-coli. In particular, walk through whatever method you have in mind for AI alignment. You get to play the role of the AI; with your sophisticated brain, massive computing power, and large-scale resources, hopefully you can satisfy the values of a few simple e-coli cells.

Perhaps you say “this is simple, they just want to maximize reproduction rate.” Ah, but that’s not quite right. That’s optimizing for the goals of the process of evolution, not optimizing for the goals of the [godshatter](https://www.lesswrong.com/posts/cSXZpvqpa9vbGGLtG/thou-art-godshatter) itself. The e-coli has some frozen-in values which have evolved to approximate evolutionary fitness maximization in some environments; your job is optimize for the frozen-in approximation, even in _new_ environments. After all, we don’t want a strong AI optimizing for the reproductive fitness of humans - we want it optimizing for humans’ own values.

On the other hand, perhaps you say “these cells don’t have any consistent values, they’re just executing a few simple hardcoded algorithms.” Well, you know what else doesn’t have consistent values? Humans. Better be able to deal with that somehow.

Perhaps you say “these cells are too simple, they can’t learn/reflect/etc.” Well, chances are humans will have the same issue once the computational burden gets large enough.

This is the problem of AI alignment: we need to both define and optimize for the values of things with limited computational resources and inconsistent values. To see the problem from the AI’s point of view, look through a microscope.
# What are some concrete problems about logical counterfactuals?

Logical counterfactuals are key to [Functional Decision Theory](https://intelligence.org/2017/10/22/fdt/) and last I heard still an unsolved problem. Unfortunately, I am still rather confused about what exactly we are trying to solve. The only concrete problem I know of in this space is the [5-and-10 problem](https://wiki.lesswrong.com/wiki/5-and-10). But as far as I know, this is solved by writing programs that immediately cause a paradox if they ever discover their output. So presumably there are some unsolved concrete problems that relate to logical counterfactuals?

**Edit**: I should mention my post on the [Cooperation Game](https://www.lesswrong.com/posts/NcA3dMJoWWEN4BQet/logical-counterfactuals-and-the-cooperation-game) as an example. Plus the further work section of this [slideshow](https://intelligence.org/wp-content/uploads/2015/05/Slepnev-Models-of-decision-making-based-on-logical-counterfactuals.pdf).
# Fifteen Things I Learned From Watching a Game of Secret Hitler

Epistemic Status: Not likely to be _true _things. Right?

1.  Liberals know nothing, fascists know everything.
2.  Most of the policies democratic governments could pass are fascist policies that expand government power.
3.  The remaining policies are liberal policies. There is no such thing as a conservative policy.
4.  Liberal policies do nothing.
5.  If the liberals do nothing enough times, they win and can congratulate themselves, no matter how much more fascist things got in the meantime.
6.  Governments must always be passing new policies, and never take away old policies. Thus, government inevitably gets more powerful over time.
7.  The more liberal policies you pass, the more likely it is any future policy will be fascist.
8.  The more fascist policies you pass, the more likely it is any future policy will be fascist.
9.  When the time comes to pass a policy, the government will choose from whatever proposals are lying around, even if all of them are fascist and everyone choosing is a liberal. There is almost never an option to just not do that, as such bold action requires a mostly fascist policy already be in place.
10.  If the government fails to agree to pass one of the things lying around, that’s even worse, because it will then choose a new policy _completely_ at random from what is lying around, which will probably be fascist.
11.  Liberals spend most of their time being paranoid over which people claiming to be liberals are secretly fascists, or even secretly actual literal Hitler, as opposed to attempting to write or choose good policies.
12.  Someone enacting liberal policies, but not in a position to assume dictatorial power, is providing strong evidence they are probably secretly Hitler.
13.  When good people often have no choice but to do bad things, but there is no way to verify this, the default is for no one who is good to have any idea who is good and who is bad.
14.  Despite this, good people _think _they know who is good and who is bad.
15.  Introducing a random element to the play is good for veteran players, because the ‘good guys’ are no longer able to (and thus forced to as in Resistance/Avalon) fall back purely on an announced, deterministic strategy, as the ‘bad guys’ could know the rules and game the system.
# In Defense of Finance

  

![](https://putanumonit.files.wordpress.com/2018/12/cosmic-suit-banker.png)

[I'm sharing the link](https://putanumonit.com/2018/12/14/defense-of-finance/) instead of a full cross-post because this essay has:

*   5,300 words
*   2 footnotes, with links that actually work
*   1 GIF, 2 charts, 5 expanding brains
*   30+ lively comments

  

This is a "much more than you wanted to know" type post on the financial industry:

*   Why everyone hates it, except for people in positions of power.
*   The value of finance in coordinating trade across space and time, told via the parable of Banksy and the corn cobs.
*   Are mortgage-backed securities a scam or a brilliant innovation?
*   The stupid arguments about bailouts, and the smarter arguments for/against bailouts.
*   Economists writing books about bank equity ratios without googling what those equity ratios actually are.
*   What _is_ up with equity ratios? Is making banks hold 50% capital the solution to financial crises?
*   What banking regulations actually make banks do.
*   The inextricable romance between finance and government.
*   A final request for incrementalism and humility.
# Equivalence of State Machines and Coroutines

In the past I often referred to the equivalence between state machines and coroutines as a kind of obvious fact that doesn't need any additional explanation. It was brought to my attention, however, that that may not always be the case.

This article therefore doesn't attempt to express and deep and ground-breaking truth, rather, it illustrates the equivalence of finite state machines and coroutines using a practical example.

The example is stolen from [Wikipedia's article on finite state machines](https://en.wikipedia.org/wiki/Finite-state_machine):

> A turnstile, used to control access to subways and amusement park rides, is a gate with three rotating arms at waist height, one across the entryway. Initially the arms are locked, blocking the entry, preventing patrons from passing through. Depositing a coin or token in a slot on the turnstile unlocks the arms, allowing a single customer to push through. After the customer passes through, the arms are locked again until another coin is inserted.

Wikipedia presents the state machine in question in the following manner:

![](http://250bpm.wdfiles.com/local--files/blog:141/fsm3.png)

And here's my attempt to rewrite the state machine as a coroutine in Go:

![](http://250bpm.wdfiles.com/local--files/blog:141/fsm2.png)
# Experiences of Self-deception

It seems to me that self-deception can describe two different things - conscious and unconscious self-deception.

Sometimes the elephant believes something untrue all by itself without the rider ever getting a look in. The claims of [elephant in the brain](http://elephantinthebrain.com/) seem to focus on this type of unconscious self-deception.

At other times the rider is complicit in endorsing a particular known untrue belief. The elephant analyses a situation, determines what it is beneficial to believe and motivates the rider to believe this. The rider has access to information which indicates that this isn't true. If the rider brings this information to full attention then it is one of those rare occasions where he can override the elephant's desires. However the rider also has the option to push the information to the side and believe a beneficial lie. It is possible to do this well enough that the information is forgotten or completely overridden with new, inaccurate, information.

In pushing the information to the side, the rider can sometimes just never bring the information to full attention. Failing that, it can drown the information out by presenting other information (which agrees with its favoured interpretation) as loudly as possible in order to doubt/ignore/forget the information which it doesn't like.

At least, this is something I experience but I don't know whether other people do. I have a few examples where this has happened and have even experimented with allowing myself to start down the route of conscious self-deception to see what it feels like. To me it feels like cognitive dissonance (feeling hot, brain feeling "fuzzy", adrenaline kicking in) whilst the rider works on counteracting the information. I guess this would be followed by the relief of resolving said dissonance when the rider starts to believe the lie but I haven't experimented that far!

The literature appears to be understandably non-committal on whether the subjects are consciously aware of their self-deception - I guess that would be pretty hard to determine.

So my question is - do other people recognise this as something which happens to them? How would you describe the experience? Is it something which you've trained yourself to recognise when it starts?
# What self-help has helped you?

Simply, what self-help techniques have you tried and found to help you? Bonus points if you can say something about the context in which you did the self-help and any speculations you have about mechanism of action.

Please give one technique per answer, multiple answers accepted per person (at least, I hope LW allows that, but I'm guessing it will), please read existing answer to avoid duplication, and comment on existing answers about a technique if you have more to say about that technique.

I'll put a couple of my own answers in below as examples and because I have answers to my own question!
# Standing on a pile of corpses

\[In the darkest day of 2018, it is proper to think about the darkness that surrounds us\]

When we think about the history of humanity, we focus on its highlights.

Galileo discovering the moons of Jupiter. Edward Jenner developing the first pox vaccine. Emmy Noether setting the mathematical foundation of modern physics.

We stand on the shoulders of giants, we say, that have elevated us over the clouds so we can see the stars above.

I believe a more apt metaphor is that we stand on a pile of nameless corpses.

Because for each great human that made it to the history books, a million have been forgotten. And each of them, knowingly or unknowingly, it is part of our legacy.

And most of them are dead.

100 billion humans have been born and died, most in undignified circumstances of sickness and age, and most by no choice of their own. They are now part of the pile of corpses.

7 billion people remain alive. And even though our lives are much better than those of the people from 1000 years ago, from 100 years ago and even from 10 years ago, we still suffer.

Many of our living kin live in sickness and hunger. Even amongst the most fortunate we wrestle with mental illness and accidents and the plights of aging.

We do not stand proud, but afraid and resigned to become yet another layer of the pile of corpses.

For not a single human lives free of the tyranny of death. We can choose to embrace it early, but we cannot still postpone it, not for much.

And so the pile of corpses grows, too fast for us to properly mourn the fallen.

There is a glimmer of hope, that the pile of corpses will stop growing as we come of age as a civilization. That we will stop the non consensual suffering we experience, and death will be no more for those who want to defy it.

That we will have, for the first time in history, time to breathe and reflect and remember all the nameless people who came before. To properly ponder without having to constantly struggle over our survival, and leaving behind extreme suffering.

To finally give a proper burial to the pile of corpses we stand on, and decide what to do with our piece of the universe.

But we are not guaranteed this happy ending - the book of humanity might suddenly end, without us having a chance to dictate the final words.

The worst we have endured is not a good predictor of the worst that it is to come, and for all we know the horrors ahead may be the ones that end us.

And then the pile of corpses will stop growing, but so will our civilization.

And all there will be left is an inanimate pile of corpses floating through the cosmos, surrounded by the cold and uncaring void.

_Thanks to Tam Borine for proofreading the text. This text was used as part of a 2018 private secular solstice celebration in Madrid, Spain._
# Anthropic probabilities and cost functions

I've claimed that anthropic probabilities like [SIA](https://en.wikipedia.org/wiki/Self-indication_assumption) and [SSA](https://en.wikipedia.org/wiki/Self-sampling_assumption) don't actually exist - or, more properly, that you need to include some details of preferences in order to get any anthropic probabilities, and thus that anthropic issues should be [approached from the perspective of decision theory](https://arxiv.org/abs/1110.6437).

What do I mean by this? Well, informally, what are probabilities? If I said that $X$ (a very visible event) would happen with a probability $10\%$, then I would expect to see events like $X$ happen about a tenth of the time.

This makes a lot of sense. Why can't it be transposed into anthropic situations? Well, the big problem is the "I" in "I would expect". Who is this "I" - me, my copies, some weighted average of us all?

In non-anthropic situations, we can formalise "I would expect to see" with a cost function. Let me choose a number $p(X)$ to be whatever I want; then, if $X$ doesn't happen I pay a cost of $(p(X))^2$, while if it does happen, I pay a cost of $(p(X)-1)^2$ (this is exactly equal to $-(p(X)-I_X)^2$, for $I_X$ the [indicator function](https://en.wikipedia.org/wiki/Indicator_function) of $X$).

Then, for this cost function, I minimize my losses by setting "$p(X)$" to be equal to my subjective opinion of the probability of $X$ (note there are many eliciting [cost functions](https://jfinocchiaro.files.wordpress.com/2018/05/cvx-elic4.pdf) we could have used, not just the quadratic loss, but the results are the same in for all of them).

In the informal setting, we didn't know how to deal with "I" when expecting future outcomes. In the formal setting, we don't know how to aggregate the cost when multiple copies could all have to pay the cost.

There are two natural methods of aggregation: the first is to keep $-(p(X)-I_X)^2$, as above, as the cost for every copy. Thus each copy has the *average* cost of all the copies (this also allows us to generalise to situations where different copies would see different things). In this case, the probability that develops from this is SSA.

Alternatively, we could add up all the costs, giving a *total* cost of $-n(p(X)-I_X)^2$ if there were $n$ copies (this also generalises to situations where different copies see different things). In this case, the probability that develops from this is SIA.

So probability might be an estimate of what I expect to see, or a cost-minimiser for errors of prediction, but anthropic probabilities differ depending on how one extends "I" and "cost" to situations of multiple copies.
# Why Don't Creators Switch to their Own Platforms?

Almost every content creator rationalists follow owns their platform: podcasters like Sam Harris and the Julia Galef, bloggers like Scott (and [myself](https://putanumonit.com/)), all the nerdy webcomics. And yet, outside the rationalsphere every creator seems engaged in an endless fight against censorship and harassment by the platforms that are supposed to enable them: Facebook, Twitter, Tumblr, Patreon... So why do they stay on those platforms? Other than Sam Harris giving Patreon the middle finger, no one else seems to do much except protest platforms on the platforms themselves.

This questions really came up for me after reading the [saga of Pewdiepie and YouTube](https://quillette.com/2018/12/20/pewdiepies-battle-for-the-soul-of-the-internet/). Currently, pewdiepie.com redirects to his YouTube page, where he posts videos protesting YouTube. This is crazy. The technology that YouTube provides was hard to build when YouTube started a decade and a half ago, but surely today it's not a huge challenge. PDP has 20 billion total views. He doesn't need traffic from the algorithm suggesting his videos, everyone else is trying to game the algorithm to get redirected by PDP! Switching to his own platform would allow him to capture a higher percentage of revenue, be immune to any kind of censorship, and make him a legend if he starts an exodus from YouTube. He can host all the other non-PC comedians on his own platform. How is that not worth losing a bit of traffic as viewers readjust?
# Best arguments against worrying about AI risk?

Since so many people here (myself included) are either working to reduce AI risk or would love to enter the field, it seems worthwhile to ask what are the best arguments against doing so. This question is intended to focus on existential/catastrophic risks and not things like technological unemployment and bias in machine learning algorithms.
# Testing Rationality Apps for Science

**TLDR**: Do you want to contribute to science by trying out new productivity apps? Then send a mail to re-experiments-on@tuebingen.mpg.de.

**Longer version**: Do you want to achieve your personal goals more effectively, become more focused and productive at work, improve your planning skills, or learn how to make better decisions?

Then we have good news for you: You can be the first to try out our brand-new productivity and self-improvement apps by participating in our research.

We regularly conduct online studies and are looking for participants willing to try out our productivity apps. You would be amongst the first to test our newly developed innovative applications that foster personal growth and/or productivity.

We are the Max Planck Research Group for Rationality Enhancement at the MPI for Intelligent Systems in Tübingen. Our scientific mission is to lay the cognitive and technological foundations for helping people become more effective. To this end, we investigate how people learn how to think, develop cognitively-informed machine learning methods for discovering rational heuristics, and build cognitive tutors that help people learn faster and cognitive prostheses that assist people in goal setting and goal achievement.

If you would like to contribute to our research and participate in our upcoming studies, please subscribe to our mailing list by sending an email to re-experiments-on@tuebingen.mpg.de.

If you can think of other people or communities who might be interested in using our apps and participating in our research, then please feel to spread the word by sharing or forwarding this message to them.

Many thanks in advance for your support!
# Reinterpreting "AI and Compute"

Some arguments saying that the recent evidence about the speed at which compute has been increasing and has been responsible for rapid progress in machine learning, might mean that we should be less worried about short timelines, not more.

> \[...\] Overall, it seems pretty common to interpret the OpenAI data as evidence that we should expect extremely capable systems sooner than we otherwise would.

> However, I think it’s important to note that the data can also easily be interpreted in the opposite direction. The opposite interpretation goes like this:

> 1\. If we were previously underestimating the rate at which computing power was increasing, this means we were o_verestimating_ the returns on it.

> 2\. In addition, if we were previously underestimating the rate at which computing power was increasing, this means that we were _overestimating_ how sustainable its growth is.

> **3\.** Let’s suppose, as the original post does, that increasing computing power is currently one of the main drivers of progress in creating more capable systems. Then — barring any major changes to the status quo — it seems like we should expect progress to slow down pretty soon and we should expect to be underwhelmed by how far along we are when the slowdown hits.
# On Disingenuity

Suppose someone claims that all morality is relative, but when pressed on whether this would apply even to murder, they act evasive and refuse to give a clear answer. A critic might conclude that this person is disingenuous in refusing to accept the clear logical consequences of their belief.

However, imagine that there's a really strong social stigma against asserting that murder might not be bad, to the point of permanently damaging such a person's reputation, even though there's no consequence for making the actually stronger claim that all morality is relative. The relativist might therefore see the critic as the one who is disingenuous; trying to leverage social pressure against them instead of arguing on the basis of reason.

Thus in the right circumstances, each side can quite reasonably see the other as disingenuous. I suspect that everyone will have experienced both sides of the coin at different times depending on the issue being discussed.

_This post was produced with the support of the [EA Hotel](http://eahotel.org/)_
# Can dying people "hold on" for something they are waiting for?

_content note: death, old age, sickness_

I've heard numerous anecdotal accounts of sick or old people who are on death's door "holding on" in a way suggestive that they were exerting some effort to do, until they had reached closure on some thing (a relative coming to visit, a manuscript published, somebody's birthday).

I could imagine this being a totally real thing that dying people can do for some limited time.

I can also imagine it just being cherry picked stories that were more a matter of luck.

It seems likely that there's at least situations where, say, eating is difficult/painful, and people continue exerting effort to do that so long as they have something that feels worth it to keep doing so, and then stop putting in the effort after hitting some milestone they cared about.

Some of the anecdotes I've heard implied something more immediate going on (where someone seemed to be holding on and literally a few minutes or seconds afterwards, died).

(Possible straightforward mechanism could just be that _breathing_ becomes painful and difficult, and people only keep doing it when they have a concrete goal)
# In what ways are holidays good?

I'd like a model of the benefits that holidays (vacations) can have, so that I can plan accordingly. Relevant questions that I have, although feel free to answer ones not listed here:

*   Do holidays teach you things about other places that you couldn't learn from Wikipedia?
*   Why are holidays more relaxing than just lying in bed at home and paying somebody else to take care of you?
*   Does visiting family count as a holiday in the relevant sense?
*   Are the benefits of tourism and/or pilgrimage the same as the benefits of holidays? What are they?
*   How much money should I be willing to spend on holidays?
*   Is 'holiday' a coherent enough category that I can treat it as primitive for the purpose of this question?
# 1987 Sci-Fi Authors Timecapsule Predictions For 2012

I find it interesting how focused people were on certain trends such as venereal disease, which turned out almost irrelevant just 25 years later as drivers of global norms. Of the set I think Algis Budrys wins Most Accurate:

>Because we will be in a trough between 20th-century resources and 21st-century needs, in 2012 all storable forms of energy will be expensive. Machines will be designed to use only minimal amounts of it. At the same time, there will be a general expectation that a practical cheap-energy delivery system is just around the corner. Individuals basing their career plans on any aspect of technology will concentrate on that future, leaving contemporary machine applications to the less ambitious or to those who foresee a different future. The most socially approved-of individuals will constitute a narrowly focused aristocracy, and will be at the mercy of dull functionaries and secretive rebels who actually perform the day-to-day maintenance of society. It should be noted that most minimal-energy devices process information and microscopic materials, not consumer goods. The function of "our" society may depend on processing information and biotechnology to subjugate goods-producing societies. These societies may be geographically external, or may be yet another social stratum within central North America. In either case, crowd-management technologies will have to turn away from forms that might in any way impair capital goods production. Social regimentation will then have become so deft that most people will regard any other social milieu as pitiable.
# What makes people intellectually active?

What is the difference between a smart person who has read the sequences and considers AI x-risk important and interesting, but continues to be primarily a consumer of ideas, and someone who starts having ideas? I am not trying to set a really high bar here -- they don't have to be good ideas. They can't be off-the-cuff either, though. I'm talking about someone taking their ideas through multiple iterations.

A person does not need to research full-time to have ideas. Ideas can come during downtime. Maybe it is something you think about during your commute, and talk about occasionally at a lesswrong meetup.

There is something incomplete about my model of people doing this vs not doing this. I expect more people to have more ideas than they do.

AI alignment is the example I'm focusing on, but the missing piece of my world-model extends much more broadly than that. How do some people end up developing sprawling intellectual frameworks, while others do not?

There could be a separate "what could someone do about it" question, but I want to avoid normative/instrumental connotations here to focus on the causal chains. Asking someone "why don't you do more?" has a tendency to solicit answers like "yeah I should do more, I'm bottlenecked on willpower" -- but I don't think willpower is the distinguishing factor between cases I observe. (Maybe there is _something related_ involved, but I mostly don't think of intellectual productivity as driven by a top-down desire to be intellectually productive enforced by willpower.)

I have some candidate models, but all my evidence is anecdotal and everything seems quite shaky.
# Thoughts on Q&A so far?

It's been a few weeks since introducing the Open Questions / Q&A features on LessWrong.

As the team returns from the holidays, we'll likely put some time into fine tuning the features and introducing supporting elements to make them work a bit better. I thought it'd be good to check in with how people were overall feeling about them now that they've seen them, and what additional features would be most useful to flesh out the system. (Either features on questions themselves, or supporting features to help keep track of questions)
# What do you do when you find out you have inconsistent probabilities?

I've recently been reading about a rationalist blogger who converted to Catholicism. She may have assigned subjective probabilities like:

P(Objective Morality)=0.9P(God)=0.05

Then she may have introspected and come up with:

P(Objective Morality | God)=0.99P(Objective Morality | No God)=0.02

 We can calculate:

P(God|Objective Morality)=P(Objective Morality|God)×P(God)P(Objective Morality)=0.99×0.050.9=0.55P(God|No Objective Morality)=P(No Objective Morality|God)×P(God)P(No Objective Morality)=0.01×0.050.1=0.005

Abbreviating Objective Morality by "OM", and "God" by "G", this state of affairs is inconsistent, because we intuitively see that:

P(G)≠P(G|OM)×P(OM)+P(G|¬OM)×P(¬OM)

To resolve it, she could either increase her subjective probability of there being a God

P(God)=P(G | OM )×P(OM)+P(G | ¬OM )×P(¬OM )=0.55×0.9+0.005×0.1=0.4955≈0.5

or she could reduce her probability of there being some kind of objective morality

P(OM)=P(OM | G )×P(G)+P(OM | ¬G )×P(¬G )=0.99×0.1+0.02×0.95=0.118

She could also reconsider P(God|Objective Morality) or P(Objective Morality|God).

* * *

Anyways, I find myself very confused by this state of affairs. Is this a solved question? Is there a purely principled way of resolving this which only takes into account the 4 numbers P(OM), P(G), P(OM|G) and P(G|OM)? Is there a standard way of using some kind of metaprobabilities?
# What exercises go best with 3 blue 1 brown's Linear Algebra videos?

3 blue 1 brown is a youtube channel that teaches math concepts. I've found it a much better introduction than other resources I've looked at. The _[Essence of Calculus](https://www.youtube.com/watch?v=WUvTyaaNkzM&list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr)_ series was particularly good.

But one issue is that, while the videos come with a few exercises sprinkled within (typically one per concept), they don't come with enough to really check whether I understand a thing.

Last year I tried binging the channel but kept running into issues where I'd want to do additional practice, so I'd try Khan Academy or Brilliant.org. But it felt like the exercises were introducing concepts in a fairly different order, or without using the same metaphors/explanations/keywords that 3-blue-1-brown was. So it was hard to tell what exercises corresponded with what. (Brilliant.org was better overall than Khan Academy but still felt like a completely different lesson plan, with somewhat less-good-explanations)

Eventually, bouncing between brilliant.org and 3 blue 1 brown resulted in me running out of steam and giving up.

I've emailed the creator about it but even if they were excited about it I imagine it'd be quite a while before anything useful came of it.

I'd be generally interested in a project of crowdsourcing exercises that correspond to individual 3-blue-1-brown videos. Since I'm currently trying to re-ignite my interest in the [Linear Algebra](https://www.youtube.com/watch?v=fNk_zzaMoSs&index=2&t=0s&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab) series I thought I'd ask specifically about that, although if people happen to have cached thoughts about other series that'd be cool.
# Events in Daily?

I'm in the process of refactoring how we handle the default posts view. The core goal was to implement some anti-spam features into it. But it highlighted some oddness in how we're currently filtering things.

I wanted to check how this might effect people on GreaterWrong, and people who use the Daily page.

_tldr: would you feel better or worse if Events showed up in the daily feed? (for reference, there seems to be roughly one event listing per two days)_

_(And, for people implementing LW readers, would it be a problem if events and meta posts appeared in the default post view?)_

Technical details:

Currently our default view filters out meta posts, event posts, and posts associated with a particular community group. But, almost all the actual views we use on lesswrong.com don't really need that default filtering:

*   The curated feed only includes thing we manually put there (so it's not especially useful to filter out meta/events/group-descriptions, since we just wouldn't put them there)
*   Same for frontpage
*   Same for sequences
*   The events view only shows events anyhow
*   User profiles currently aren't showing events / meta / group posts, and this is actually a problem – it means it's harder for a user to find all their posts
*   Recent discussion already deliberately includes all these things (but has to go out of it's way to

The only two views where we actually would want to exclude things are in "All Posts" and "Daily", which serve similar functions.

"Event" posts are location-specific and so aren't as often relevant to people, but also, if your primary way of engaging with site is on /daily, you might not otherwise get to see events.

I'm curious if people who primarily read daily would feel better or worse about Events getting included there. (Again, the frequency seems to be about 1 listing per two days)
# Electrons don’t think (or suffer)

There is an EA fringe that talks about suffering in elementary particles. Physicist Sabine Hossenfelder reminds her readers why this panpsychist idea is nonsense.

TL;DR: "if you want a particle to be conscious, your minimum expectation should be that the particle can change. It’s hard to have an inner life with only one thought. But if electrons could have thoughts, we’d long have seen this in particle collisions because it would change the number of particles produced in collisions."

The whole post and the comments are an entertaining read.
# What's the best way for me to improve my English pronounciation?

I'm a native German speaker and learned a good portion of my English not in school but by reading English content.

As part of a New Years event we did a Doom Circle and one essential feedback I got was that I have to improve my English accent.

When it comes to my budget I'm willing to invest a few hundred Euro but not thousands.

What would you recommend to me to improve my English pronounciation?
# What is a reasonable outside view for the fate of social movements?

_Epistemic status: very hand-wavy and vague, but confident there is a substantial and well-understood core. Hoping for an answer that elucidates that core more clearly._

It is something of a rationalist folk theorem that social movements face the risk of an "Eternal September", or of scaling into oblivion. (See e.g. [this blog post by Leverage research](http://leverageresearch.org/blog/ea-failure-scenarios), [this paper by Owen Cotton-Barratt](http://globalprioritiesproject.org/wp-content/uploads/2015/05/MovementGrowth.pdf), [David Chapman](https://meaningness.com/geeks-mops-sociopaths) or [Benjamin Hoffman](http://benjaminrosshoffman.com/construction-beacons/) on "Geeks, MOPs and sociopaths", and Scott Alexander on ["the toxoplasma of rage"](https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/)).

I've had the sense that some cocktail of Hansonian/Dunbarian evolutionary psychology, basic game theory/microeconomics, memetic theory and [Sturgeon's law](https://en.wikipedia.org/wiki/Sturgeon%27s_law), should predict this. That is, that some reasonably operationalised version of the claim "most social movements fail" is true.

Yet I am not able to point to >=5 historical examples of social movements that suffered this fate, along with some gears for what went wrong.

Hence I'm looking for links, historical examples, more fleshed-out gears, ... anything that might form a more rigorous reference point for _an outside view on the fate of social movements_.
# Does anti-malaria charity destroy the local anti-malaria industry?

The usual argument against foreign aid to Africa is that randomly giving tons of free goods (such as food) _ruins local producers_; and when at some later moment the charity goes out of fashion (or decides to target a different part of Africa), the local situation becomes _even worse than before_, because the local producers have gone out of business. In addition, it hurts the local people psychologically to know that _any_ local business, no matter how successful it could otherwise have been, can at any moment be destroyed by a well-meaning foreign charity.

[Recently](https://www.lesswrong.com/posts/rwjv8bZfSuE9ZAigH/act-of-charity#w3W8HvKEhxRtaBqhk) I heard the same argument made about anti-malaria nets recommended by GiveWell. If I understand it correctly, the donated nets put local net producers out of business (increasing local poverty and dependence on foreign aid), and the estimated number of lives saved is misleading (because in the alternative scenario, the same people could have been saved by locally produced nets).

I have one specific question, and one more general concern.

The specific question... well, I know _nothing_ about the anti-malaria industry in Africa. It exists, I assume. But _quantitatively_ \-\- how many nets it produces, how many nets it stops producing because it is pushed out of the market by GiveWell, whether the nets are of comparable quality, what is the best estimate of the scenario with no foreign aid compared to the scenario with foreign aid -- I have no idea. I supposed some of this was _already discussed_ by some effective altruists, so I would love to hear the summary.

The meta concern is the following: I find the argument of foreign goods disrupting local market plausible. But seems to me that the problem is with _high variance_ (one year a ton of goods, the very next year nothing), not with foreign goods _per se_. Because, anytime a country participates in foreign trade, the local producers of the stuff that is being imported, are pushed out of business. But we have the law of comparative advantages saying that in global, this is a _good_ thing, for both countries. (Or to put it differently, trade _sanctions_ are typically used as a punishment, not as a reward.) I worry that at some moment, the "stop destroying African economy by your disruptive aid" argument becomes effectively "stop trading with Africa", and I am not sure where exactly to draw that line.
# Supervising strong learners
by amplifying weak experts

Abstract
--------

> Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017b), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

* * *

_Tomorrow's AI Alignment Forum sequences post will be 'AI safety without goal-directed behavior' by Rohin Shah,_ in the sequence on Value Learning.

The next post in this sequence on Iterated Amplification will be 'AlphaGo Zero and capability amplification', by Paul Christiano, on Tuesday 8th January.
# Imitation learning considered unsafe?

ETA: This post can basically be read as arguing that imitating human decisions, or any other outputs from an (approximate) planning process, seems especially likely to produce mesa-optimization, since a competent imitator should recover an (approximate) planning (i.e. optimization) process.

This post states an observation which I think a number of people have had, but which hasn't been written up (AFAIK). I find it one of the more troubling outstanding issues with a number of proposals for AI alignment.

1) Training a flexible model with a reasonable simplicity prior to imitate (e.g.) human decisions (e.g. via behavioral cloning) should presumably yield a good approximation of the process by which human judgments arise, which involves a planning process.

2) We shouldn't expect to learn exactly the correct process, though.

3) Therefore imitation learning might produce an AI which implements an unaligned planning process, which seems likely to have instrumental goals, and be dangerous.

**Example:** The human might be doing planning over a bounded horizon of time-steps, or with a bounded utility function, and the AI might infer a version of the planning process that doesn't bound horizon or utility.

**Clarifying note:** Imitating a human is just one example; the key feature of the human is that the process generating their decisions is (arguably) well-modeled as involving planning over a long horizon.

**Counter-argument(s):**

*   The human may have privileged access to context informing their decision; without that context, the solution may look very different
*   Mistakes in imitating the human may be relatively harmless; the approximation may be good enough
*   We can restrict the model family with the specific intention of preventing planning-like solutions

**Overall, I have a significant amount of uncertainty about the significance of this issue, and I would like to see more thought regarding it.**