Harley-ml/lesswrong huggingface dataset:
# On Abstract Systems

We've all seen those abstract systems that are used for analysis like: [Strategy = Ends + Ways + Means](https://ssi.armywarcollege.edu/pubs/parameters/issues/Winter_2016-17/10_Meiser.pdf), [Waterfall Model](https://www.toolsqa.com/software-testing/waterfall-model/): Requirements, System Design, Implementation, Integration & Testing, Deployment, Maintenance, [SWOT](https://www.investopedia.com/terms/s/swot.asp): Strengths, Weaknesses, Opportunities, Threats, ect.

Classes that teach these tend to be boring. Often you'll get three different business models thrown at you with minimal motivation. You won't be told about the environment this model evolved in, nor why the particular elements were included. Sometimes you'll get a concrete example, sometimes not, but if so, it's more likely to be a made up scenario rather than an example from someone's real life experience. It is even rarer that you will receive multiple such examples.

This is largely a result of people's bias towards explicit knowledge. But the explicit knowledge is in most cases simply one of many ways of carving up possibility space. Much more important is the implicit knowledge of how to apply the system and what situations it is helpful in. Strategy = Ends + Ways + Means becomes more applicable when you learn that it became dominate after the Vietnam War when it was felt that the loss was the result of attempting to achieve certain objectives without sufficient resources. So instead of just asking, "What are we trying to achieve?" and "How could we achieve it?", you also wanted to ask, "What resources are needed to achieve it?". Similarly, the Waterfall Model becomes more useful when you are told that many projects have not been as successful as desired because of a failure to consider requirements (ie. using a framework that requires Internet Explorer 10+ when some staff are stuck on Internet Explorer 9). Similarly, people often perform the cost analysis of projects for just the initial coding, without accounting for the maintenance cost. This is a system for making sure that you don't forget the obvious.

It would be even better if I could illustrate these with examples from my own experience, but this should still be sufficient to illustrate my point. I do think that these systems can be useful. But only when communicated in the right way.

_This post was written with the support of the [EA Hotel](http://eahotel.org/)_
# Open Thread January 2019

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), and [seeing if there are any meetups in your area](https://www.lesswrong.com/community).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# 
LW Update 2019-1-09 – Question Updates, UserProfile Sorting

First update of 2019!

Streamlined Mobile Above-the-Fold
---------------------------------

On mobile, logged out users now see cleaner, slightly sleeker collections of Rationality: A-Z, Codex and HPMOR. Logged in users see only one recommended sequence instead of 3.

Improved Q&A Features
---------------------

We have a few quality-of-life improvements to the new question system:

*   You can now move a comment into the answers section, if you later realize it was a pretty good answer.
*   Answers that appear on the frontpage are now styled differently from comments.
*   If you reply to an answer on the frontpage, on the corresponding Question page it now appears parented underneath the answer as intended.

UserProfile and All Posts Sorting
---------------------------------

On User Profile pages, as well on the beta-testing [All Posts](https://lesswrong.com/allPosts) page, there's now a reasonable method of sorting posts.

You can now sort posts by:

*   new
*   old
*   top (highest karma)
*   magic (our karma/time weighted algorithm)
*   recentComments (most recently commented posts sorted to the top, along with new posts)

You also can filter posts so you only see Curated, Frontpage, Questions, Meta and Event. (By default, User Profiles, All Posts and Daily will show all post types)

GreaterWrong now gets better spam protection
--------------------------------------------

We had previously implemented a filter so that new user's posts didn't appear on the home page until approved by an admin. But this only applied to specific frontpage post views. Now, it applies to all views by default, so GreaterWrong should get less spam in their All Posts page.

Password Resetting
------------------

Resetting passwords should now work properly (previously it required an awkward workaround where you had to log out before you could reset it)
# What shape has mindspace?

Epistemological status: Babbling.

Let  a map each mind in mindspace to how aligned it is. We are trying to optimize a. To that end, lemmata are helpful which talk about the [shape](https://ncatlab.org/nlab/show/stuff,+structure,+property) of mindspace. That's why we try to call it a space even before defining what category C it lives in.

To optimize a function, start with a diverse enumeration of its domain. The deontological enumeration covers all others with constant-factor overhead, but the consequentialist enumeration gives us more properties to work with.

Every mind m has an implicit utility function u(m). a factors through u as a function, but not as a continuous function, let alone a C-morphism. That's why we've recently moved away from explicit utility maximizers.

Use mathematical language to tell our story! Then we might guess where it's going.
# Why is so much discussion happening in private Google Docs?

I've noticed that when I've been invited to view and comment on AI-safety related draft articles (in Google Docs), they tend to quickly attract a lot of extensive discussion, including from people who almost never participate on public forums like LessWrong or AI Alignment Forum. The number of comments is often an order of magnitude higher than a typical post on the Alignment Forum. (Some of these are just pointing out typos and the like, but there's still a lot of substantial discussions.) This seems kind of wasteful because many of the comments do not end up being reflected in the final document so the ideas and arguments in them never end up being seen by the public (e.g., because the author disagrees with them, or doesn't want to include them due to length). So I guess I have a number of related questions:

1. What is it about these Google Docs that makes people so willing to participate in discussing them?

2. Would the same level of discussion happen if the same draft authors were to present their drafts for discussion in public?

3. Is there a way to attract this kind of discussion/participants to public posts in general (i.e., not necessarily drafts)?

4. Is there some other way to prevent those ideas/arguments from "going to waste"?

5. I just remembered that LessWrong has a sharable drafts feature. (Where I think the initially private comments can be later made public?) Is anyone using this? If not, why?

Personally I much prefer to comment in public places, due to not wanting my comments to be "wasted", so I'm having trouble understanding the psychology of people who seem to prefer the opposite.
# Does freeze-dried mussel powder have good stuff that vegan diets don't?

Personal background for this question:
- I'm a vegan for ethical reasons, meaning that I don't want to contribute to the farming of beings that can suffer.
- I think that oysters and mussels probably aren't sentient.
- Oysters and mussels seem pretty disgusting to me.
- I would like to be healthy.
- I am already taking B12 and Omega-3 supplements.

Oysters and mussels seem like they might have important nutrients that one otherwise can't get in a vegan diet without supplementation, just because that animals are different to plants/fungi/etc., and humans ate animals in the EEA. However, due to their disgustingness, I can't quite bring myself to eat them.

Luckily, it turns out that one can [buy](https://www.amazon.com/gp/product/B008DS2BL2/) capsules that contain freeze-dried green-lipped mussel powder. I'm pretty excited about this prospect, but am somewhat worried that the freeze-drying and powderifying process would destroy various nutrients, or render them mysteriously ineffective in the ways that I gather vitamin pills are mysteriously less effective than vitamins in food. I also imagine that maybe the really important nutrients are only found in (say) land mammals. Can anybody shed light on this?
# this Website uses Bayesian networks to evaluate claims - RootClaim

I found out this website, [RootClaim](https://www.rootclaim.com), about a year and a half ago (a bit after it was founded), much before finding LW. that's how i found out about Bayesian probability, and i remember getting crazy at the potential of it :)

So i now thought "surly someone posted about it on LW" - i searched, and found nothing. I wonder if there's a reason or it's just that somehow no one here heard about it.

Anyway, it uses Bayesian trees (a special case of Bayesian networks), to evaluate different hypothesis.

anyone can sumbit a topic. and then anyone can submit hypothesis, sources, and so on. everything is crowdsourced.

as a tool, it seems to have a lot of potential, and i even saw that Nissim Taleb (Author of the black swan) twitted about it favorably.

main problem is, it seems to be quite inactive, both the community (which is super small), and the website itself. the last update was in December 2017...

I would love to hear what you guys think about it, or the potential of something like it :)

here's the link again: [https://www.rootclaim.com](https://www.rootclaim.com)
# What are the open problems in Human Rationality?

LessWrong has been around for 10+ years, CFAR's been at work for around 6, and I think there have been at least a few other groups or individuals working on what I think of as the "Human Rationality Project."

I'm interested, especially from people who have invested significant time in attempting to push the rationality project forward, what they consider the major open questions facing the field. (More details in [this comment](https://www.lesswrong.com/posts/gX8fcAwk3HGkFyJgk/what-are-the-open-problems-in-human-rationality#MPcuH6dWEDi8TDo96))

**"What is the Rationality Project?"**

I'd prefer to leave "Rationality Project" somewhat vague, but I'd roughly summarize it as _"the study of how to have optimal beliefs and make optimal decisions while running on human wetware."_

If you have your own sense of what this means or should mean, feel free to use that in your answer. But some bits of context for a few possible avenues you could interpret this through:

Early LessWrong focused a lot of cognitive biases and how to account for them, as well as Bayesian epistemology.

CFAR (to my knowledge, roughly) started from a similar vantage point and eventually started moving in the direction of "how to do you figure out what you actually want and bring yourself into 'internal alignment' when you want multiple things, and/or different parts of you want different things and are working at cross purposes. It also looked a lot into Double Crux, as a tool to help people disagree more productively.

CFAR and Leverage both ended up exploring introspection as a tool.

Forecasting as a field has matured a bit. We have the Good Judgment project.

Behavioral Economics has begun to develop as a field.

I recently read "How to Measure Anything", and was somewhat struck at how it tackled prediction, calibration and determining key uncertainties in a fairly rigorous, professionalized fashion. I could imagine an alternate history of LessWrong that had emphasized this more strongly.

With this vague constellation of organizations and research areas, gesturing at an overall field...

...what are the big open questions the field of Human Rationality needs to answer, in order to help people have more accurate beliefs and/or make better decisions?
# When is CDT Dutch-Bookable?

I gave a [rough sketch of a Dutch Book against CDT,](https://www.alignmentforum.org/s/fgHSwxFitysGKHH56/p/wkNQdYj47HX33noKv) with an example. I expect it can be turned into a fairly general theorem. What setting should we use to formalize this? What are the necessary assumptions? What's the general formulation of the Dutch Book? Are there interesting cases where this fails? How does it relate to anthropic decision problems, and anthropic Dutch-books?
# What are the components of intellectual honesty?

I have a pretty strong intuition that "intellectual honesty" points to a specific thing, with a characteristic set of behaviors (both outward (in a conversation/writing) and inward (patterns of thinking)). My concept of intellectual honesty also seems to largely coincide with the concept as used by others, but I'm not sure.

I am talking about a high standard. This isn't at all about violating a code of academic ethics.

I am talking about the sort of thing which helps to foster [epistemic trust](http://lo-tho.blogspot.com/2014/12/epistemic-trust.html), helping to [take conversations to higher levels](https://www.lesswrong.com/posts/WB49uKgMkQRbKaHme/combat-vs-nurture-and-meta-contrarianism). (But don't read those links if you already know enough what I mean to answer and want to avoid anchoring your view to mine.)
# Is there a.. more exact.. way of scoring a predictor's calibration?

On platforms like PredictionBook, a user's credibility will be measured by looking at a histogram of all of their predictions (edit: After being told about "brier score", I have become able to notice that PredictionBook totally shows a brier score right under the histogram. Still, thinking about brier scores, they're not very good either. You can only maximise your brier score by betting on things you're certain of, it creates perverse incentives to avoid placing bets on uncertain events, which we already have enough of in natural discourse). For a perfectly well calibrated agent, exactly 90% of their 0.9 predictions should have come true, exactly 20% of their 0.2 predictions should have come true, etc. We can reduce a person's calibration to a number by, I suppose, getting the sum of squares of the differences of each histogram from its middle point (I don't know if that's how they do it).

But whatever they do with the histogram, I think I'll find it very unsatisfying, because the result are going to depend on what spacing we used for the histogram segments, and that's arbitrary. We could have a histogram with ten segments, or a histogram with fifty segments, we'll get different scores. I feel like there must be some exact.. continuous way of scoring a predictor's calibration, and that the math will probably be generally useful for other stuff. Is there a method?
# Buy shares in a megaproject

I think we should create a separate legal construct for megaprojects.

I recently wrote about megaproject management [here](https://www.lesswrong.com/posts/XBQXDcGs3NhqRscxH/megaproject-management). In the outline for Oxford's [Major Programme Management](https://www.sbs.ox.ac.uk/programmes/msc-major-programme-management/msc-major-programme-management-outline) program, the first module is described in this way:

> Develop your understanding of major programmes as a governance structure and distinctive organisational form. In the context of programme performance, consider and reflect on organisational theory and design.

The basic concept is that instead of a normal corporation, it would operate like a corporation with an expiration date and the project goals baked into the governance. We want "on-time and on-budget" to take the place of "growth and revenue" in the new organization's decision making.

*   We can buy stock in corporations, which [now live ~16 years](https://research-doc.credit-suisse.com/docView?language=ENG&format=PDF&sourceid=em&document_id=1070991801&serialid=TqtAPA%2FTEBUW%2BgCJnJNtlkenIBO4nHiIyPL7Muuz0FI%3D) and declining.
*   Megaprojects like the [Big Dig](https://en.wikipedia.org/wiki/Big_Dig) (16 years of construction) and the [F-35](https://en.wikipedia.org/wiki/Lockheed_Martin_F-35_Lightning_II) development (26 years to production) are easily in the range of normal corporations
*   Currently the only way to bet for or against the megaproject is to bet on the corporations/creditors/etc.
*   This would allow us to specifically invest in, or bet against, the project.
*   We invest in time-limited financial assets like bonds, puts and options already.

Consider the [Boring Company](https://www.boringcompany.com/projects/). If one of those projects is a terrible idea, we are stuck with adjusting our position on the company as a whole, which means the terrible project puts the others in jeopardy.

By contrast, if the megaprojects were all separate entities and the Boring Company was hired to _manage_ all of them, we could then bet on Boring Company and all of the projects independently of each other.

It seems to me that this would do a pretty good job resolving a lot of the problems that plague megaprojects currently.
# In SIA, reference classes (almost) don't matter

This is [another](https://www.lesswrong.com/posts/jH3NfxoNgKTh9r5KW/anthropics-full-non-indexical-conditioning-fnc-is) write-up of a fact that is generally known, but that I haven't seen proven explicitly: the fact that [SIA](https://en.wikipedia.org/wiki/Self-indication_assumption) does not depend upon the reference class.

Specifically:

* Assume there are a finite number of possible universes $U_i$. Let $R$ be a reference class of finitely many agents in those universes, and assume you are in $R$. Let $R_s$ be the reference class of agents subjectively indistinguishable from you. Then SIA using $R$ is independent of $R$ as long as $R_s \subset R$.

Proof:

Let $\{U_i\}_{i\in I}$ be a set of universes for some indexing set $I$, and $P$ a probability distribution over them. For a universe $U_i$, let $R(U_i)$ be the number of agents in the reference class $R$ in $U_i$.

Then if $P_R$ is the probability distribution from SIA using $R$:
* $P_R(U_i) = \frac{R(U_i) P(U_i)}{\sum_{j\in I} R(U_j)P(U_j)}$.

We now wish to update on our own subjective experience $sub$. Since there are $R(U_i)$ agents in our reference class, and $R_s(U_i)$ have subjectively indistinguishable experiences, this updates the probabilities by weights $P_R(U_i) \to P_R(U_i) \times \frac{R_s(U_i)}{R(U_i)}$, which is just $R_s(U_i) P(U_i)/\left(\sum_{j\in I} R(U_j)P(U_j)\right)$. After normalising, this is:

$\begin{array}{rl}
P_R(U_i|sub) &= \frac{P(U_i) R_s(U_i)}{\sum_{k\in I} P(U_k) R_s(U_k)} \times \frac{\left(\sum_{j\in I} R(U_j)P(U_j)\right)}{\left(\sum_{j\in I} R(U_j)P(U_j)\right)}\\
&= \frac{P(U_i) R_s(U_i)}{\sum_{k\in I} P(U_k) R_s(U_k)} \\
&= P_{R_s}(U_i).
\end{array}$

Thus this expression is independent of $R$.

Given some measure theory (and measure theoretic restrictions on $R$ to make sure expressions like $\int_I R(U_i)P(U_i)$ converge), the result extends to infinite classes of universes, with $\int$ in the proof instead of $\sum$.
# Solving the Doomsday argument

The [Doomsday argument](https://en.wikipedia.org/wiki/Doomsday_argument) gives an anthropic argument for why we might expect doom to come reasonably soon. It's known that the Doomsday argument works under [SSA](https://en.wikipedia.org/wiki/Self-sampling_assumption), but not under [SIA](https://en.wikipedia.org/wiki/Self-indication_assumption).

Ok, but since different anthropic probability theories are [correct answers to different questions](https://www.lesswrong.com/posts/nxRjC93AmsFkfDYQj/anthropic-probabilities-answering-different-questions), what are the question versions of the Doomsday argument, and is the original claim correct?

## No Doomsday on birth rank

Simplify the model into assuming there is a large universe (no Doomsday any time soon) with many, many future humans, and a small one (a Doomsday reasonably soon - within the next 200 billion people, say), with equal probability. In order to think in terms of frequencies, which comes more naturally to humans, we can imagine running the universe many, many times, each with the $50\%$ Doomsday chance.

There are roughly a [108.5 billion humans](https://www.prb.org/howmanypeoplehaveeverlivedonearth/) who have ever lived. So, asking:

* What proportion of people with birth rank 108.5 billion, live in a small universe (with a Doomsday reasonably soon)?

The answer to that question converges to $1/2$, the SIA probability. Half of the people with that birth rank live in small universes, half in large universes.

## Doomsday for time travellers

To get an SSA version of the problem, [we can ask](https://www.lesswrong.com/posts/wdEGA3RM7ESqHXKMX/the-questions-and-classes-of-ssa):
 
* What proportion of universes, where a randomly selected human has a birthrank of 108.5 billion, will be small (with a Doomsday reasonably soon)?

This will give an answer close to $1$ as it converges on the SSA probability.

But note that this is generally *not* the question that the Doomsday argument is posing. If there is a [time traveller](https://upload.wikimedia.org/wikipedia/en/8/80/Versions_of_the_Doctor.jpg) who is choosing [people](https://en.wikipedia.org/wiki/Companion_(Doctor_Who)) at random from amongst all of space and time - then if they happen to choose you, that is a bad sign for the future (and yet another reason you should go with them). Note that this is consistent with [conservation of expected evidence](https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence): if the time traveller is out there but doesn't choose you, then this a (very mild) update towards no Doomsday.

But for the classical non-time-travel situation, the Doomsday argument fails.
# EA Funds: Long-Term Future fund is open to applications until Feb. 7th

(Crossposted [to the EA Forum](https://forum.effectivealtruism.org/posts/oFeGLaJ5bZBBRbjC9/ea-funds-long-term-future-fund-is-open-to-applications-until) feel free to comment on either post with any questions)

The Long-Term Future Fund doing another round of grants in mid-February (currently ~$400k), and are open to applications until February 7th. You can apply [here](https://docs.google.com/forms/d/e/1FAIpQLSeDTbCDbnIN11vcgHM3DKq6M0cZ3itAy5GIPK17uvTXcz8ZFA/viewform?usp=sf_link). (Excerpt from the application:)

> We are particularly interested in small teams and individuals that are trying to get projects off the ground, or that need less money than existing grant-making institutions are likely to give out (i.e. less than ~$100k, but more than $10k). Here are a few examples of project types that we're open to funding an individual or group for (note that this list is not exhaustive):

> \+ To spend a few months (perhaps during the summer) to research an open problem in AI alignment or AI strategy and produce a few blog posts or videos on their ideas

> \+ To spend a few months building a web app with the potential to solve an operations bottleneck at x-risk organisations

> \+ To spend a few months up-skilling in a field to prepare for future work (e.g. microeconomics, functional programming, etc).

> \+ To spend a year testing an idea that has the potential to be built into an org.

> You can find more details on the kind of project we are likely to fund on the fund page: [https://app.effectivealtruism.org/funds/far-future](https://app.effectivealtruism.org/funds/far-future) If we don’t feel that we have enough information but we think your project sounds promising we may be in touch for more information.
# The Very Repugnant Conclusion

I haven't seen the [very repugnant conclusion](https://pdfs.semanticscholar.org/110e/2189a930e4de714b81db2c137b2548b9bcca.pdf) mentioned much here, so I thought I'd add it, as I need it as an example in a [subsequent post](https://www.lesswrong.com/posts/msJexF6sxojRHRLbi/synthesising-divergent-preferences-an-example-in-population).

Basically, the repugnant conclusion says:

* Let $W_1$ be a world filled with very happy people leading meaningful lives. Then, according to total utilitarianism, there is a world $W_2$ which is better than $W_1$, where everyone has lives barely worth living - but the population is huge.

Some people come to accept the repugnant conclusion, sometimes reluctantly. More difficult to accept is the very repugnant conclusion:

* Let $W_1$ be a world filled with very happy people leading meaningful lives. Then, according to total utilitarianism, there is a world $W_2$ which is better than $W_1$, where there is a population of suffering people much larger than the total population of $W_1$, and everyone else has lives barely worth living - but the population is very huge.

This one feels more negative than the standard repugnant conclusion, maybe because it strikes at our [egalitarian](https://en.wikipedia.org/wiki/Egalitarianism) and [prioritarian](https://en.wikipedia.org/wiki/Prioritarianism) instincts, or maybe because of the nature of suffering.

Anyway, my motto on these things is generally:

* When you find morally wrong outcomes that contradict your moral theory, then enrich your moral theory rather than twisting your moral judgements.
# How could shares in a megaproject return value to shareholders?

In my [previous post](https://www.lesswrong.com/posts/exw3RJJbsPEmXvMFC/buy-shares-in-a-megaproject) on buying shares in a [megaproject](https://www.lesswrong.com/posts/XBQXDcGs3NhqRscxH/megaproject-management), several people asked the most reasonable of questions: how would the shares return value?

A simple description of shareholder value:

Assets - Liabilities = Equity

Basic project outcome:

Budget - Costs = Surplus

So the concept is fundamentally to treat the budget as the assets column, put the costs in the liability column, and treat every dollar you are under the budget as an increase in shareholder equity. At the conclusion of the project, the shares will be bought back for that surplus in cash.

This means the price the share sells for is a direct bet on how well the project will do against the budget.

This is analogous to how investors buy in to start-ups; the investment becomes the assets and in exchange they get a share of the equity. But in the case of megaprojects, the initial investor is getting the project outcome as compensation; the equity should instead go to management as their compensation. Now this gives management the incentive to be as efficient as possible, while _also_ giving them the flexibility to raise more capital in exchange for equity.

For comparison with some of the ways this is currently done, consider the [fixed-price](https://en.wikipedia.org/wiki/Fixed-price_contract) contract and the [cost-plus](https://en.wikipedia.org/wiki/Cost-plus_contract) contract. The former gives the contractor an incentive to be efficient by forcing them to hold all the risk; the latter shifts the risk back to the contractee. Fixed-price is common for things that are predictable, like services; cost-plus is common for things that are risky or require R&D to complete, like defense procurement. Megaprojects are usually of the latter sort.

Tangential but worth also addressing: why a financial asset instead of a prediction market?

*   Financial assets provide direct incentives, whereas prediction markets provide information.
*   Financial assets are a built-in reference class; a megaproject asset would naturally be compared against all other megaproject assets.
*   Financial assets have a huge market, and there are established methods for making bets for/against/on/with them. There are no large prediction markets, so an entire market would have to be built. The problem of incentivizing the new market seems more difficult than the one of incentivizing a new type of asset.
# Doing Despite Disliking: Self‐regulatory Strategies in Everyday Aversive Activities

A large survey of self-regulatory strategies and how reportedly effective they are.

Much more in the typical "self-regulatory" paradigm for self-help psychology, where people are assumed to have a lot more control over the strategies they choose (i.e. downplaying interactions between actions and attitudes), but I'm curious what people's thoughts are.
# What math do i need for data analysis?

I always had fairly good mathematical thinking (I think) and loved learning about beautiful concepts in math - but i didn't learn much at all in school (cause i had the choice). You can say i was "utilitarian" regarding learning math, i didn't do it if i didn't see how it can enrich my life.

so my knowledge of math is quite disorganized, i know more about Bayes theorem then many much simpler concepts (i know, it really shouldn't be that way).

Now i want to be able to analyze data, but i don't want to learn math that i won't use for it, if possible.

So here's my question - what basic stuff do i need to learn in order to be able to calculate probabilities, statistics, do Bayesian math, and overall do things within data analysis that I may yet be aware of?

If you also have suggestions for how to learn _those_ things, after i learn the basics, it will be much appreciated.

thank you :)
# Why not tool AI?

An extremely basic question that, after months of engaging with AI safety literature, I'm surprised to realize I don't fully understand: why not tool AI?

AI Safety scenarios seem to conceive of AI as an autonomous agent. Is that because of the current machine learning paradigm, where we're setting the AI's goals but not specifying the steps to get there? Is this paradigm the entire reason why AI safety is an issue?

If so, is there a reason why advanced AI would need an agenty utility function sort of set up? Is it just too cumbersome to give step by step instructions for high level tasks?

Thanks!
# Announcement: AI alignment prize round 4 winners

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the fourth round of the [AI Alignment Prize](https://www.lesswrong.com/posts/juBRTuE3TLti5yB35/announcement-ai-alignment-prize-round-3-winners-and-next), funded by Paul Christiano. From July 15 to December 31, 2018 we received 10 entries, and are awarding four prizes for a total of $20,000.

## The winners

We are awarding two first prizes of $7,500 each. One of them goes to Alexander Turner for [Penalizing Impact via Attainable Utility Preservation](https://www.lesswrong.com/posts/mDTded2Dn7BKRBEPX/penalizing-impact-via-attainable-utility-preservation); the other goes to Abram Demski and Scott Garrabrant for the [Embedded Agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh) sequence.

We are also awarding two second prizes of $2,500 each: to Ryan Carey for [Addressing three problems with counterfactual corrigibility](https://www.lesswrong.com/posts/owdBiF8pj6Lpwwdup/addressing-three-problems-with-counterfactual-corrigibility), and to Wei Dai for [Three AI Safety Related Ideas](https://www.alignmentforum.org/posts/vbtvgNXkufFRSrx4j/three-ai-safety-related-ideas) and [Two Neglected Problems in Human-AI Safety](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety).

We will contact each winner by email to arrange transfer of money. Many thanks to everyone else who participated!

## Moving on

This concludes the AI Alignment Prize for now. It has stimulated a lot of good work during its year-long run, but participation has been slowing down from round to round, and we don't think it's worth continuing in its current form.

Once again, we'd like to thank everyone who sent us articles! And special thanks to Ben and Oliver from the LW2.0 team for their enthusiasm and help.
# Game Analysis Index

This post links to this blog’s posts discussing game design, balance, economics and related topics, as well as any strategy posts. It does not contain new content.

Much of the blog is relevant to gaming, but these are the explicitly on-topic posts.

#### Eternal Sequence

[Eternal, and Hearthstone Economy versus Magic Economy](https://thezvi.wordpress.com/2018/02/10/eternal-and-hearthstone-economy-versus-magic-economy/)

[The Eternal Grind](https://thezvi.wordpress.com/2018/03/30/the-eternal-grind/)

[Eternal: The Exit Interview](https://thezvi.wordpress.com/2018/10/10/eternal-the-exit-interview/)

#### Artifact / Card Rebalancing Sequence

[Review: Artifact](https://thezvi.wordpress.com/2018/11/22/review-artifact/)

[Artifact Embraces Card Balance Changes](https://thezvi.wordpress.com/2018/12/26/artifact-embraces-card-balance-changes/)

[Card Collection and Ownership](https://thezvi.wordpress.com/2018/12/27/card-collection-and-ownership/)

[Card Balance and Artifact](https://thezvi.wordpress.com/2018/12/28/card-balance-and-artifact/)

[Card Rebalancing, Card Oversupply and Economic Considerations in Digital Card Games](https://thezvi.wordpress.com/2018/12/31/card-rebalancing-and-economic-considerations-in-digital-card-games/)

[Advantages of Card Rebalancing](https://thezvi.wordpress.com/2019/01/01/advantages-of-card-rebalancing/)

[Disadvantages of Card Rebalancing](https://thezvi.wordpress.com/2019/01/06/disadvantages-of-card-rebalancing/)

#### Game Reviews (Including Those Listed Above)

All games reviewed are recommended, we don’t generally waste time on unworthy games.

[Persona 5: Spoiler-Free Review](https://thezvi.wordpress.com/2017/06/13/persona-5-spoiler-free-review/)

[Review: Artifact](https://thezvi.wordpress.com/2018/11/22/review-artifact/)

[Review: Slay the Spire](https://thezvi.wordpress.com/2018/12/09/review-slay-the-spire/)

[Octopath Traveler: Spoiler-Free Review](https://thezvi.wordpress.com/2018/11/05/octopath-traveler-spoiler-free-review/)

#### Magic Strategy

[Deck Guide: Burning Drakes](https://thezvi.wordpress.com/2018/11/13/deck-guide-burning-drakes/)
# Should questions be called "questions" or "confusions"  (or "other")?

_Or, what is the best way to think about "what is a question?" on LW?_

The LW Team just had a retreat where we thought through a lot of high level strategy. We have a lot of ideas building off of the "questions" feature.

One thing that struck us is that a lot of early stage research has less to do with formalizable questions, and more to do with noticing anomalies in your current model/paradigm. Something feels off that you can't explain, or there's a concept you don't even understand well enough to ask a coherent question about.

The "question" feature was meant, in part, to reduce the cost of exploring early stage curiosity, but we wondered if it might even be a slightly-too-formalized.

Just like, technically there was nothing stopping you from asking a question as a post (but adding the feature caused a proliferation of questions) there is nothing stopping you from asking an ill-formed question. But, maybe changing the language slightly would better encourage early-stage curiosity.

So, curious:

How would feel if we changed "Ask a question" to "Pose a confusion" or something like that? (The main issue so far is that "pose confusion" is, well, way more confusing since it's a non-standard phrase. Other options include literally saying "Ask question/Pose Confusion" \[i.e. both at once, so you get the benefit of the clear-cut "ask question"\], or some word other than "pose.")

(Somewhat but not-entirely-jokingly, we also noticed people are hesitant to post "answers" since they sound like you're trying to claim you know what you're talking about. We jokingly considered "Post a deconfusion", or "post a partial answer" as options)
# Link: That Time a Guy Tried to Build a Utopia for Mice and it all Went to Hell

Video: [https://www.youtube.com/watch?v=5m7X-1V9nOs](https://www.youtube.com/watch?v=5m7X-1V9nOs)

Text version: http://www.todayifoundout.com/index.php/2018/12/that-time-a-guy-tried-to-build-a-utopia-for-mice-and-it-all-went-to-hell/

"In 1968, an expert on animal behaviour and population control called John B. Calhoun built what was essentially a utopia for mice that was purpose built to satisfy their every need. Despite going out of his way to ensure the inhabitants of his perfect mouse society never wanted for anything, within 2 years virtually the entire population was dead. So what happened?"
# Shanghai LessWrong Meetup

（[中文翻译](http://hpmor.lofter.com/post/382652_12d80c733)）

I'm inviting to start a new Shanghai LW meetup, with focus on a broader discussion about less wrong in the AI safety and cryptography in context of the long-term outcomes for life and humanity. I'd suggest the common theme of the meetup to be [composition of relations](https://en.wikipedia.org/wiki/Composition_of_relations) in general sense, — e.g., compositions of neuron models, compositions of cryptographic functions, etc.

The initial objective of this meetup may well be to know the readers of LessWrong in Shanghai, and exchange the ideas how to estimate the collection of everyone's true (minimally biased) utility functions, and find ways to optimize the composition of relations (world's I/O) systems to improve the way they satisfy them.

**Suggested meetup content:** self-introductions describing what types of problems of the world (sub-optimalities) everyone personally cares to address, and kind of ideas or systems they are working on to bring about those changes. There will be a big screen, where we can exchange and share presentations of thoughts related.

Fragments of the meetup content will later be shared on [https://safe-ai.org](https://safe-ai.org).

* * *

**Where**: China, Shanghai, Putuo, 富丽大厦 (Thanks to [@Tedko](https://www.lesswrong.com/users/Tedko) for the venue!)  
**When**: 2019-02-06 14:00~21:00 (UTC+08).

The reason for long period from midday to evening, is because it is a kind of exploratory post. The last Shanghai LW Meetup I've seen [was organized](https://www.lesswrong.com/posts/JXb2JWGpFamHdxayt/meetup-shanghai-less-wrong-meetup-0) around 2012, so, and there seem to have been no Shanghai LW Meetups for quite a while.
# Is Agent Simulates Predictor a "fair" problem?

It's a simple question, but I think it might help if I add in context. In the paper introducing [Functional Decision Theory](https://arxiv.org/pdf/1710.05060.pdf), it is noted that it is impossible to design an algorithm that can perform well on all decision problems since some of them can be specified to be blatantly unfair, ie. punish every agent that isn't an alphabetical decision theorist.

The question then arises, how do we define which problems are or are not fair? We start by noting that some people consider Newcomb's-like problems to be unfair since your outcome depends on a predictor's prediction, which is rooted in an analysis of your algorithm. So what makes this case any different from only rewarding the alphabetical decision theorist?

The paper answers that the prediction only depends on the decision you end up making and that any other internal details are ignored. So it only cares about your decision and not how you come to it, the problem seems fair. I'm inclined to agree with this reasoning, but a similar line of reasoning doesn't seem to hold with Agent Simulates Predictor. Here the algorithm you use is relevant as the predictor can only predict the agent if it's algorithm is less than a certain level of complexity, otherwise it may make a mistake.

Please note that this question isn't about whether this problem is worth considering; life is often unfair and we have to deal with it the best that we can. The question is about whether the problem is "fair", where I roughly understand "fair" meaning that this is in a certain class of problems that I can't specify at this moment (I suspect it would require its own seperate post) where we should be able to achieve the optimal result in each problem.
# For what do we need Superintelligent AI?

Most of human problems could be solved by humans or slightly above human AIs. Which practically useful tasks – which we could define now – require very high level of superintelligence? I see two main such tasks:

1) Prevention of the creation of other potentially dangerous superintelligences.

2) Solving the task of indefinite life extension, which include solving aging and uploading.

I could imagine other tasks, like near-light-speed space travel, but they are neither urgent nor necessary.

For which other tasks do we need superintelligence?
# Prediction Contest 2018: Scores and Retrospective

Way back in April 2018, I [announced a Prediction Contest](https://www.lesswrong.com/posts/sYW9oMXhLHCvaCoLX/prediction-contest-2018), in which the person who made the best predictions on a bunch of questions on PredictionBook ahead of a 1st July deadline would win a prize after they all resolved in January 2019, which is now.

It was a bit of an experiment; I had no idea how many people were up for practicing predictions to try to improve their calibration, and decided to throw a little money and time at giving it a try. And in the spirit of reporting negative experimental results: The answer was 3, all of which I greatly appreciate for their participation. I don't regret running the experiment, but I'm going to pass on running a Prediction Contest 2019. I don't think this necessarily rules out trying to practically test and compete in rationality-related areas in other ways later, though.

The Results
-----------

Our entrants were bendini, bw, and Ialaithion, and their ranked log scores were:

bw: -9.358221122

Ialaithion: -9.594999615

bendini: -10.0044594

This was sufficiently close that changing a single question's resolution could tip the results, so they were all pretty good. That said, bw came out ahead, and even managed to beat averaging everyone's predictions- if you simply took the average prediction (including non-entrants) as of entry deadline and made that your prediction, you'd have got -9.576568147.

The full calculations for each of the log scores, as well as my own log score and the results of feeding the predictions as of prediction time to a simple model rather than simply averaging them, are [in a spreadsheet here](https://docs.google.com/spreadsheets/d/11EWTrR0EfPzVklMx25C-woA1C7GVbNeP4ikgEdaXRYI/edit?usp=sharing).

I'll be in touch with bw to sort out their prize this evening, and thanks to everyone who participated and [who helped with finding questions](https://www.lesswrong.com/posts/5iNxs98Bq2rxMkc6g/2018-prediction-contest-propositions-needed) to use for it.
# "The Unbiased Map"

“Long have we suffered under the tyranny of maps.

Biased maps which show topography, but not population.

Wretched maps which speak of religion, but not languages.

Divisive maps which paint with the color of Party, but not the color of economic conditions.

Dirty maps which show crop yield across the heartland, but neglect Fiber Optic Internet coverage.

Our age calls for better maps, maps free from the bias of these old maps, perfect maps.

Imagine the day of the unbiased map. The map which shows both how to get to the airport via public transit and GDP by county. The holy map demonstrating last year’s rainfall and the distribution of seminaries and rabbinical schools. The ancestral map depicting migration of immigrants and American tribes in 1491.

Don’t give me an atlas which pretends at perfection but hisses red herrings from hydra-heads. I want the real thing, a map which doesn’t end at some arbitrary border whether it be the county line, or the sphere of earth. A map which can show the world as known by the Qing Dynasty, Strabo, Majorcan Jews, and the Aztecs. A map of Elon Musk’s neurons and a map of the solar system.

Today’s maps enlighten as the Brothers Grimm, through a bundle of fairy tales. There are no ethical maps under capitalism, all of them drip with nullification and interposition. None show me the world that should be, none provide directions to Valhalla, all show but some thin surface of Reality. And for Mankind, the surface does not satisfy!”
# A small example of one-step hypotheticals

Just a small example of what [one-step hypotheticals](https://www.lesswrong.com/posts/i6hWWcKyxBPj7ELT6/one-step-hypothetical-preferences) might mean in theory and in practice.

This involves a human H pricing some small object:

![](https://www.dropbox.com/s/27ci25golftth76/mini_violin.jpg?raw=1)


## In theory

The human H is (hypothetically) asked various questions that causes it to model how much they would pay for the small violin. These questions are asked at various times, and with various phrasings, and the results look like this:

![](https://www.dropbox.com/s/kbvdxub9mt8ngbv/val.png?raw=1)

Here the costings are all over the place, and one obvious way of reconciling them would be to take the mean (indicated by the large red square), which is around 5.5.

But it turns out there are extra patterns in the hypotheticals $\mathcal{H}_t$ and the answers $f(\mathcal{H}_t)$. For example, there is a clear difference between valuations that are done in the morning, around midday, or in the evening. And there is a difference if the violin is (accurately) described as "handmade"L

![](https://www.dropbox.com/s/a9lnnxqcw6s0byt/val_detailed.png?raw=1)

There are now more options for finding a "true" valuation here. The obvious first step would be to over-weight the evening valuations, as there are less datapoints there (this would bring the average up a bit). Or one could figure out whether the "true" H was better represented by their morning, midday, or evening selves. Or whether their preference for "handmade" objects was strong and genuine, or a passing positive affect. H's various meta-preferences would all be highly relevant to these choices.

# In practice

Ok, that's what might happen if the agent had the power to ask unlimited hypothetical questions in arbitrarily many counterfactual scenarios. But that is not the case in the real world: the agent would be able to ask one, or maybe two questions at most, before the human attitude to the violin would change, and further data would become tainted.

Note that if the agent had a good brain model of H, it might be able to simulate all the relevant answers; but we'll assume for the moment that the agent doesn't have the capabilities.

So, in theory, huge amounts of data and many relevant patterns that are meta-preferentially relevant. In practice, two values maximum.

Now, if this was all that the agent had access to, then it could only use a crude guess. But if the agent was investigating the human more thoroughly, it could do a lot more. The pattern of valuing things differently at different times of the day might show up over longer observations, as would the pattern of reacting to key words in the description. If the agent assumed that "valuing objects" was not something that humans did ex-nihilo with each object (with each object having its own independent quirky biases), then it could apply the template across all valuations, and from even a single data point (along with knowledge of the time of day, the description, etc...) come up with an estimate that was closer to the theoretical one.