Harley-ml/lesswrong huggingface dataset:
# A Brief History of LessWrong


In 2006, [Eliezer Yudkowsky](https://www.lesswrong.com/users/eliezer_yudkowsky), [Robin Hanson](https://www.lesswrong.com/users/robin_hanson2), and others began writing on _[Overcoming Bias](https://www.overcomingbias.com/about)_, a group blog with the general theme of how to move one’s beliefs closer to reality despite biases such as overconfidence and wishful thinking. In 2009, after the topics drifted more widely, Eliezer moved to a new community blog_, LessWrong_.

LessWrong was seeded with series of daily blog posts written by Eliezer, originally known as _The Sequences_, and more recently compiled into an edited volume, _[Rationality: A-Z](https://www.lesswrong.com/rationality)._ These writings attracted a large community of readers and writers interested in the art of human rationality.

In 2015-2016 the site underwent a steady decline of activity leading some to declare the site dead. In 2017, a team led by Oliver Habryka took over the administration and development of the site, relaunching it on an [entirely new codebase](https://github.com/LessWrong2/Lesswrong2) later that year.

The new project, dubbed LessWrong 2.0, was the first time LessWrong had a full-time dedicated development [team](https://www.lesswrong.com/posts/aG74jJkiPccqdkK3c/the-lesswrong-team-page-under-construction) behind it instead of only volunteer hours. Site activity recovered from the 2015-2016 decline and [has remained at steady levels](https://www.lesswrong.com/posts/9dA6GfuDca3Zh3RMa/data-analysis-of-lw-activity-levels-age-distribution-of-user) since the launch.

The team behind LessWrong 2.0 has ambitions not limited to maintaining the original LessWrong community blog and forum. The LessWrong 2.0 team conceives of itself more broadly as an organization attempting to build community, culture, and technology which will drive intellectual progress on the world’s most pressing problems.
# "But It Doesn't Matter"

If you ever find yourself saying, "Even if Hypothesis _H_ is true, it doesn't have any decision-relevant implications," _you are rationalizing!_ The fact that _H_ is interesting enough for you to be considering the question at all (it's not some arbitrary trivium like the 1923th binary digit of π, or the low temperature in São Paulo on September 17, 1978) means that it must have some relevance to the things you care about. It is _vanishingly improbable_ that your optimal decisions are going to be the _same_ in worlds where _H_ is true and worlds where _H_ is false. The fact that you're tempted to _say_ they're the same is probably because some part of you is afraid of some of the imagined consequences of _H_ being true. But _H_ is already true or already false! If you happen to live in a world where _H_ is true, and you make decisions as if you lived in a world where _H_ is false, you are thereby missing out on all the extra utility you would get if you made the _H_-optimal decisions instead! If you can figure out exactly what you're afraid of, maybe that will help you work out what the _H_-optimal decisions are. Then you'll be a [better position to successfully notice](https://www.lesswrong.com/posts/3XgYbghWruBMrPTAL/leave-a-line-of-retreat) which world you _actually_ live in.
# Welcome and Open Thread June 2019

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), and [seeing if there are any meetups in your area](https://www.lesswrong.com/community).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# I translated 'Twelve Virtues of Rationality' into Hebrew.

[Here it is](https://docs.google.com/document/d/1Y9TfVw_-w1vDP5hE39twq9d9qekuEkA8In0h8GW0_NE/edit?usp=sharing) \- if you know Hebrew and have feedback, do give it, either in the comments here or in the document (it's not fully edited).

I don't really know what am i supposed to do with it now, though. where should i put it? can/should i put it on my future website? (of course i don't own the translation anymore than the original piece)

  

Regarding the translation, the thing I'm most uncertain of is (ironically) which word to use for 'virtue', if you have a better idea, please share.
# What is the evidence for productivity benefits of weightlifting?

_\[Mod Note: This question received an answer that seemed worth curating. See the answer by LW user [hereisonehand](https://www.lesswrong.com/users/hereisonehand) for the curation notice\]_

I've been weightlifting for a while, and I've heard vaguely good things about it's effect on productivity, like a general increase in energy levels. A recent quick google search session came up empty. If someone looks into the literature and finds something interesting I'll pay a $50 prize.*

Assume the time horizon is <5 years. I'd prefer answers focus predominantly on productivity benefits. Effects on cardiovascular could be part of an analysis, but would not qualify on their own. If the evidence is for something clearly linked to productivity, like sleep, I'd count that. Introspective evidence will also not qualify. Comparisons to other forms of exercise would be especially interesting. Assume a healthy individual, although I'm at least somewhat interested in effects on individuals with depression or anxiety given their prevalence.

*Prize to go to best answer, as judged by me, if there are any that meet some minimal threshold of rigor, also as judged by me.
# How is Solomonoff induction calculated in practice?

Solomonoff induction is generally given as the correct way to penalise more complex hypotheses when calculating priors. A great introduction can be found [here](https://www.lesswrong.com/posts/Kyc5dFDzBg4WccrbK/an-intuitive-explanation-of-solomonoff-induction).

My question is, how is this actually calculated in practice?

As an example, say I have 2 hypotheses:

A. The probability distribution of the output is given by the same normal distribution for all inputs, with mean  μ and standard deviation σ.

B. The probability distribution of the output is given by a normal distribution depending on an input x with mean μ0+mx and standard deviation σ.

It is clear that hypothesis B is more complex (using an additional input \[x\], having an additional parameter \[m\] and requiring 2 additional operations to calculate) but how does one calculate the actual penalty that B should be given vs A?
# All knowledge is circularly justified

Many philosophers have tried to find the foundations of our knowledge, but why do we think there are any? The framing of foundations implies a separate bottom layer of knowledge from which everything is built up. And while this is undoubtedly a useful model in many contexts, why should we believe in this as the complete and literal truth as opposed to merely a simplification?

Consider:

1) If we dig deep enough into any of our truth claims, we'll eventually reach a point at which they are justified by intuition

2) The reliability of intuition or various intuitions is not something that is merely taken as basic or for granted, but can instead be justified somewhat by arguments from experience and evolutionary arguments.

3) However both empirical verification and evolutionary arguments themselves both rely on assumptions that are justified by intuition

This is circular, but is this necessarily a problem? If your choice is a circular justification or eventually hitting a level with no justification, then the circular justification suddenly starts looking pretty attractive. In other words, we have to be comparative and consider what the alternative is to circular epistemology and not just consider it in isolation.

Is this important? It seems to depend on context. For applied rationality, not so much. But I would like to suggest that the more philosophical areas of the rationalist project would look quite different if they were built upon a circular epistemology.
# Arbital scrape

Update: [Arbital Scrape V2](https://www.lesswrong.com/posts/8pSh54GoJJqfHdndk/arbital-scrape-v2)

I've scraped [http://arbital.com](http://arbital.com) as the site is unusably slow and hard to search for me.

The scrape is locally browsable and plain HTML save for MathJax.

[https://drive.google.com/open?id=1b7dKhOzfMpFwngAeI8efeOzv147Lv_mx](https://drive.google.com/open?id=1b7dKhOzfMpFwngAeI8efeOzv147Lv_mx)

[https://emma-borhanian.github.io/arbital-scrape/](https://emma-borhanian.github.io/arbital-scrape/)

If there's interest let me know as I may tidy up and open source my code. edit: working on this

![](https://i.imgur.com/kNMI9GJ.png)

Mirror: [www.obormot.net/arbital](https://www.obormot.net/arbital)
# Visiting the Bay Area from 17-30 June

I'll be in the Bay Area from the 17-30 June for training as a TripleByte Interviewer. My interests include:

*   Effective altruism - particularly movement building
*   Agent foundations - particularly updateless decision theory and anthropics
*   Infinite ethics
*   More broadly, the boundary of philosophy of maths, but also philosophy in general

PM me if you want to meet up or if you have any recommendations.

Update: I'll be training during weekdays, so I'll only be free during evenings and weekends
# AGI will drastically increase economies of scale

In [Strategic implications of AIs’ ability to coordinate at low cost](https://www.lesswrong.com/posts/gYaKZeBbSL4y2RLP3/strategic-implications-of-ais-ability-to-coordinate-at-low), I talked about the possibility that different AGIs can coordinate with each other much more easily than humans can, by doing something like merging their utility functions together. It now occurs to me that another way for AGIs to greatly reduce coordination costs in an economy is by having each AGI or copies of each AGI profitably take over much larger chunks of the economy (than companies currently own), and this can be done with AGIs that don't even have explicit utility functions, such as copies of an AGI that are all corrigible/[intent-aligned](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6) to a single person.

Today, there are many industries with large economies of scale, due to things like fixed costs, network effects, and reduced deadweight loss when monopolies in different industries merge (because they can internally charge each other prices that equal marginal costs), but because coordination costs among humans increase super-linearly with the number of people involved (see [Moral Mazes and Short Termism](https://www.lesswrong.com/posts/2Zsuv5uPFPNTACwzg/moral-mazes-and-short-termism) for a related recent discussion), that creates diseconomies of scale which counterbalance the economies of scale, so companies tend to grow to a certain size and then stop. But an AGI-operated company, where for example all the workers are AGIs that are intent-aligned to the CEO, would eliminate almost all of the internal coordination costs (i.e., all of the coordination costs that are caused by value differences, such as all the things described in Moral Mazes, "market for lemons" or lost opportunities for trade due to [asymmetric information](https://en.wikipedia.org/wiki/Information_asymmetry), principal-agent problems, monitoring/auditing costs, costly signaling, and suboptimal Nash equilibria in general), allowing such companies to grow much bigger. In fact, from purely the perspective of maximizing the efficiency/output of an economy, I don't see why it wouldn't be best to have (copies of) one AGI control everything.

If I'm right about this, it seems quite plausible that some countries will foresee it too, and as soon as it can feasibly be done, nationalize all of their productive resources and place them under the control of one AGI (perhaps intent-aligned to a supreme leader or to a small, highly coordinated group of humans), which would allow them to out-compete any other countries that are not willing to do this (and don't have some other competitive advantage to compensate for this disadvantage). This seems to be an important consideration that is missing from many people's pictures of what will happen after (e.g., intent-aligned) AGI is developed in a slow-takeoff scenario.
# Learning magic

Magic (primarily misdirection and cold-reading, but also the mechanics like sleight-of-hand) seems like an extremely good case-study in the study of how the human mind works and the predictable ways in which human maps differ from the territory. There are several magicians out there who offer to teach classes and so forth, but are there any who can be vouched for as really "knowing their stuff" as a teacher if I wanted to approach the subject in this light? Relatedly, can anyone vouch for the quality of the Penn and Teller course on the MasterClass platform?

Bonus points for teachers in London who can be hired for small (up to 10 people) groups.
# On pointless waiting

I’ve often noticed in myself a tendency, if I am not doing something immediately engrossing, to find myself waiting.

Waiting, waiting, waiting, not really being present, just willing time to pass.

But the weird thing is, frequently there isn’t anything in particular that I’m waiting _for_. Getting out of that situation, yes, but I don’t have anything in particular that I’d want to do when I do get out.

I have a suspicion that this might have to do with mental habits ingrained in school.

In elementary school, there’s no real _goal_ for your studies. Mostly it’s just coming there, doing the things that teachers want you to do, until the day is over and you get to go.

In that environment, every minute that passes means winning. Every minute takes you a bit closer to being out of there. That’s the real goal: getting out so you can finally do something fun.

During a lesson you are waiting for recess, during recess you are waiting for the end of the day. Outside school you are waiting for the weekend, on the weekend you are waiting for the bliss of the long summer leave.

Waiting, waiting, waiting.

So you learn to pay attention to the time. Human minds are tuned to feedback, things that let them know how well they are doing. And since each passing minute takes you closer to the goal, the passing of time becomes its own reward.

Time having passed means that you have achieved something. Time having passed means that you can feel a tiny bit of satisfaction.

And then that habit, diligently trained for a decade, can carry over to the rest of your life. Even as an adult, you find yourself waiting, waiting, waiting.

You don’t know what it is that you are waiting for, because you are not really waiting for anything in particular. Even if it would actually be more pleasant to stay engaged with the present moment, you keep tracking the time. Because waiting feels like winning, and every passing minute feels like it takes you closer to your goal.

Even if you don’t actually know what your goal is. Even if reaching your goal will only give you a new situation where you can again wait, so that you are never actually present.

Still, you keep waiting, waiting, waiting.

—

_(**[typical mind fallacy](https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizing-from-one-example)**_ _employed for the sake of artistic license; I am describing my own experience, without claiming this to be a universal one)_
# Tal Yarkoni: No, it's not The Incentives—it's you

Neuroscientist Tal Yarkoni denounces many of his colleagues' tendency to appeal to publish-or-perish incentives as an excuse for sloppy science (October 2018, ~4600 words). Perhaps read as a complement to our [recent](https://www.lesswrong.com/posts/45mNHCMaZgsvfDXbw/quotes-from-moral-mazes) [discussion](https://www.lesswrong.com/posts/2Zsuv5uPFPNTACwzg/moral-mazes-and-short-termism) of _Moral Mazes_?
# MLU: New Blog!

I'm in the process of moving mindlevelup from Wordpress to a new static site hosted by Netlify. I like this because now I have more control over scripts and the visuals. It also finally puts into place my goal of having a place to have short/longform posts.

The plan is to slowly update posts and incrementally update the site.

The new site is [here](https://mlu.red/).

My short-form blog, Muse, has also been moved [here](https://mlu.red/muse/).

RSS feeds for both main-line blog posts and short-form posts can be found the [About](https://mlu.red/about.html) page.
# What kind of thing is logic in an ontological sense?

The existence of logic seems somewhat mysterious. It's this thing that seems to exist, but unlike other things that exist, it doesn't seem to exist anywhere in specific or in any tangible form. Further, while it is easy to mock Plato for mysticism when he posits perfect forms existing in some kind of mysterious Platonic Realm, that's actually uncomfortably close to a description of what logic is often seen as.
# Can we use ideas from ecosystem management to cultivate a healthy rationality memespace?

**Background: ecosystems management practices for improving community memespaces**

One can model individual human minds, as well as a community of minds, as an “ecosystem” of “memes”. These memes might be things like:

*   Bayesian epistemology
*   a habit of checking Facebook when one wakes up
*   wiggling one’s fingers to indicate agreement with a statement
*   prefacing things one says with “My model of this is that...”
*   Doing calibration training
*   Referring to blog posts in conversation
*   Taking silent pauses to think mid-conversation

Etc. etc.

Calling this set an “ecosystem”, seems to me to be mechanistically very close to what’s actually going on. At least, this is because:

1.  Memes mutate as they are transmitted between minds
2.  Memes undergo selection pressure as they are transmitted
3.  The underlying topology/geography of social, cultural and geographical networks of people influence their spread
4.  Memes can be in equilibrium with other memes
5.  Memes can act as “invasive species”

Now there is an emerging European rationality community, largely driven by efforts from the Prague rationalists. This community imports many memes from the Bay area rationality community. For a high-level, historical examination of this memespace, see [Julia Galef’s map of bay area memespace](https://juliagalef.com/2017/02/20/map-of-bay-area-memespace/).

At a recent CFAR workshop, we discussed how we can ensure that this interaction is successful.

Five of us, felt that the ecosystem model carried sufficient mechanistic similarity to the actual situation that it would be helpful to read up on things like: protocols for deliberate introductions of new species into new environments, invasive species regulations and protection programs, pest control, and more.

**Collection of background notes**

We spent 1h researching this. Now the outside view predicts that if we were to leave it at that, the 16-page Google doc would never be used again. Hence we’re experimenting with releasing our notes together with a LessWrong question, in order to allow others to benefit from and build on our progress.

You can find our notes [here](https://docs.google.com/document/d/13bGtCz7zwZ00qYWd1xgipKCXh0JIDgNM9kneWkdA2cQ/edit?usp=sharing).

These notes are provided “as is”. I briefly went through them to make them more readable, but apart from that this should not be interpreted as something the authors endorse as being true, and despite originating at a CFAR workshop it is not official CFAR content.

**Open questions**

We’d be interested in using further research to answer questions such as:

*   What are warning signs of a memespace/ecosystem being harmed?
*   What are best practices for introducing a new meme into a memespace, and what can we learn from actual ecosystems?
*   What are some useful models for thinking about this problem?
# Storytelling and the evolution of human intelligence

This is a notice of a recent paper that may be of interest here, ["The storytelling arms race: origin of human intelligence and the scientific mind"](https://www.nature.com/articles/s41437-019-0214-2) by Enrico Coen. It is a more specific working out of the [Machiavellian Intelligence](https://en.wikipedia.org/wiki/Machiavellian_intelligence) hypothesis, taking storytelling as the means through which the deception arms race operated, and which (it suggests) also raised up language alongside intelligence. It also discusses the relationship between storytelling (where, as in war, truth is the first casualty) and science (where truth is the goal).

Disclosure: I work closely with the author, although not on the subject of this paper.
# Reasonable Explanations

Today I watched a friend do calibration practice and was reminded of how wide you have to cast your net to get well-calibrated 90% confidence. This is true even when the questions aren't gotchas, just because you won't think of all the ways something could be wildly unlike your quick estimate's model. Being well-calibrated for 90% confidence intervals (even though this doesn't sound all _that_ confident!) requires giving lots of room even in questions you really do know pretty well, because you will _feel_ like you really do know pretty well when in fact you're missing something that wrecks you by an order of magnitude.

Being miscalibrated can _feel like_ "if it were outside of this range, I have just... no explanation for that" - and then it turns out there's a completely reasonable explanation.

Anyway, I thought a fun exercise would be describing weird situations we've encountered that turned out to have reasonable explanations. In initial descriptions, present only the information you (and whoever was thinking it over with you) _remembered to consider at the time_, then follow up in [ROT-13](https://rot13.com) with what made the actual sequence of events come clear.
# Is "physical nondeterminism" a meaningful concept?

Background: Our mental models of the universe can contain uncertainty or probability-links, as in a causal network. One may have a deterministic understanding of a phenomenon if the probability-values are all 0 and 1.

Question: Beyond that, is it meaningful to distinguish whether or not the \*universe itself\* is deterministic or nondeterministic?

For example, is it meaningful to say that the Copenhagen interpretation of QM implies a "nondeterministic universe", while Many Worlds implies a "deterministic universe"?
# Does the _timing_ of practice, relative to sleep, make a difference for skill consolidation?

It is well known that sleep (both mid-day naps and nighttime sleep) has a large effect on the efficacy of motor skill acquisition. Performance on a newly learned task improves, often markedly, following a period of sleep.

A few citations (you can find many more by searching "motor skill acquisition sleep" or similar in google scholar) :

*   [https://www.nature.com/articles/nn1959](https://www.nature.com/articles/nn1959)
*   [https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000341](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000341)
*   [https://link.springer.com/article/10.1111/j.1479-8425.2012.00576.x](https://link.springer.com/article/10.1111/j.1479-8425.2012.00576.x)

I want to know if the \_timing\_ of practice, relative to sleep, makes a difference for skill acquisition.

For instance, if you practice a skill at 7:00 PM, shortly before a night of sleep, will your performance be better in the morning than if you had practiced at 7:00 AM had a full day of wakefulness, and \_then\_ gone to sleep? If so, what is the effect size?

Josh Kaufman makes a claim to this effect in his book, _The First 20 Hours_. I have no particular reason to doubt him, 40 minutes of searching on google scholar did not turn up any papers about the importance of sleep and practice timing.

Can you point me at a relevant citation?
# Does scientific productivity correlate with IQ?

Anders Ericsson, in his popular book _Peak: Secrets from the New Science of Expertise_, claims that though one may need a reasonably high IQ to even be a scientist, after that level, IQ is irrelevant to scientific success.

> The average IQ of scientists is certainly higher than the IQ of the general population, but among scientists, there is no correlation between IQ and scientific productivity. Indeed, a number of noble prize winning scientists have had IQs that would not even qualify them for Mensa, an organization who's members must have a measured IQ of at least 132, a number that put's you in the upper 2 percentile of the population.

(From chapter 8: "But What About Natural Talent", of _Peak: Secrets from the New Science of Expertise_)

My understanding is that this is completely wrong, and the best scientists tend to have higher IQs, at all levels of performance. But this is a background belief that I haven't concretely verified.

Is there a canonical reference regarding the impact of IQ on scientific contribution?
# Is there a guide to 'Problems that are too fast to Google'?

It seems to me like problems come in a variety of required response speeds, but there's a natural threshold to distinguish fast and slow: whether or not you can Google it. The slow ones, like getting an eviction notice from your landlord or a cancer diagnosis from your doctor, can't be _ignored_ but you have time to figure out best practices before you act. Fast ones, like getting bit by a rattlesnake or falling from a high place, generally require that you already know best practices in order to properly implement them.

Also useful would be the meta-guide, which just separates out which problems are fast and slow (or how fast they are). Getting bit by a tick, for example, seems like it might be quite urgent when you discover one biting you, but isn't; you have about 24 hours from when it first attaches to remove it, which is plenty of time to research proper removal technique. Getting a bruise might seem like you have time, but actually applying cold immediately does more to prevent swelling than applying cold later does to reduce it.

Of course, this is going to vary by region, profession, age, sex, habits, and so on. I'm sort of pessimistic about this existing at all, and so am interested in whatever narrow versions exist (even if it's just "here's what you need to know about treating common injuries to humans"). Basic guides also seem useful from a 'preventing illusion of transparency' perspective.
# In physical eschatology, is Aestivation a sound strategy?

In [this paper](https://arxiv.org/abs/1705.03394), Anders Sandberg, Stuart Armstrong and Milan M. Cirkovic argue that

> If a civilization wants to maximize computation it appears rational to aestivate until the far future in order to exploit the low temperature environment: this can produce a 1030 multiplier of achievable computation.

Later Charles H. Bennett, Robin Hanson, C. Jess Riedel [disagree](https://arxiv.org/abs/1902.06730), claiming

> In fact, while this assumption may apply in the distant future, our universe today contains vast reservoirs and other physical systems in non-maximal entropy states, and computer-generated entropy can be transferred to them at the adiabatic conversion rate of one bit of negentropy to erase one bit of error. This can be done at any time, and is not improved by waiting for a low cosmic background temperature. Thus aliens need not wait to be active. As Sandberg et al. do not provide a concrete model of the effect they assert, we construct one and show where their informal argument goes wrong.

Who was right?
# Preference conditional on circumstances and past preference satisfaction

I've [mentioned conditional preferences before](https://www.lesswrong.com/posts/ic8yoGBMYLtaJkbxZ/conditional-meta-preferences). These are preferences that are dependent on facts about the world, for example "I'd want to believe X if there are strong argument for X".

But there is another type of preference that is conditional: my tastes can vary depending on circumstances and on my past experience. [For example](https://www.lesswrong.com/posts/i6hWWcKyxBPj7ELT6/one-step-hypothetical-preferences#sqekDsnDs3D9Bpm5n), I might prefer to eat apples during the week and oranges on weekends. Or, [because of the miracle of boredom](https://www.lesswrong.com/posts/WMDy4GxbyYkNrbmrs/in-praise-of-boredom), I might prefer oranges if (but only if) I've been eating apples all week so far.

What if I currently want apples, would want oranges tomorrow, but falsely believe (today) that I would want apples tomorrow? This is a known problem with "[one-step hypotheticals](https://www.lesswrong.com/posts/i6hWWcKyxBPj7ELT6/one-step-hypothetical-preferences)", and a strong argument **in practice** for assessing preferences over time rather than at a single moment $t$.

In theory, there are meta-preferences that allow one to get this even at a single moment $t$, such as "I want to be able to follow my different tastes at different times" or a more formalised desire for variety and exploration.
# Discussion Thread: The AI Does Not Hate You by Tom Chivers

This post is provided as convenient place to discussion of the new book, [The AI Does Not Hate You](https://www.amazon.com/Does-Not-Hate-You-Superintelligence-ebook/dp/B07K258VCV) by Tom Chivers, which covers LessWrong and rationalist community.

> **The AI Does Not Hate You:** **Superintelligence, Rationality and the Race to Save the World**

> This is a book about AI and AI risk. But it's also more importantly about a community of people who are trying to think rationally about intelligence, and the places that these thoughts are taking them, and what insight they can and can't give us about the future of the human race over the next few years. It explains why these people are worried, why they might be right, and why they might be wrong. It is a book about the cutting edge of our thinking on intelligence and rationality _right now_ by the people who stay up all night worrying about it.

Note that the book is available on Kindle only to people with UK/European Amazon accounts. I was able to order it in physical copy in the US, but I haven't received a shipping notification yet.
# Is the "business cycle" an actual economic principle?

E.g. [Investopedia: Business Cycle](https://www.investopedia.com/terms/b/businesscycle.asp) has a section "Stages of the Business Cycle":

> All business cycles are characterized by several different stages, as seen below.

> **1\. Expansion.** This is the first stage. When expansion occurs, there is an increase in employment, incomes, production, and sales. People generally pay their debts on time. The economy has a steady flow in the money supply and investment is booming.

> **2\. Peak.** The second stage is a peak when the [economy](https://www.investopedia.com/articles/investing/042915/5-industries-driving-us-economy.asp) hits a snag, having reached the maximum level of growth. Prices hit their highest level, and economic indicators stop growing. Many people start to restructure as the economy's growth starts to reverse.

> **3\. Recession:** These are periods of contraction. During a recession, unemployment rises, production slows down, sales start to drop because of a decline in demand, and incomes become stagnant or decline.

> **4\. Depression:** [Economic growth](https://www.investopedia.com/articles/economics/12/okuns-law.asp) continues to drop while unemployment rises and production plummets. Consumers and businesses find it hard to secure credit, trade is reduced, and bankruptcies start to increase. Consumer confidence and investment levels also drop.

> **5\. Trough:** This period marks the end of the depression, leading an economy into the next step: recovery.

> **6\. Recovery:** In this stage, the economy starts to turn around. Low prices spur an increase in demand, employment and production start to rise, and lenders start to open up their credit coffers. This stage marks the end of one business cycle.

If all you know is that [markets are anti-inductive](https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-are-anti-inductive), and therefore anticipate a random walk with an average of 10%/year growth, don't you automatically expect the stock market fluctuations to sometimes look like bull markets and recessions?

So does the "business cycle" add any explanatory power, or is it merely the type of retroactive non-explanation that you get when you read a news article about the 1-day movement of the stock market?

Maybe you can argue that the business cycle is self-reinforcing due to psychological reactions.

But to the degree that the business cycle is a separate understandable phenomenon, can't investors use that understanding to place bets which make them money while dampening the effect?
# LW Dev question: FB-style tagging?

As LW is an intellectual \*\*community\*\* where people get to know each other and know "who knows what, who's working on what" it seems like something like tagging people FB-style in specific posts/comments is a very low-hanging fruit. (Of course you should be able to silence people from tagging you or sort taggers by some score). Is this in the pipeline, or has been ruled out for some reason?
# Is your uncertainty resolvable?

I was chatting with Andrew Critch about the idea of [Reacts on LessWrong](http://lesswrong.com/posts/SDELNzyboMpZpDwSg/fb-discord-style-reacts).

Specifically, the part where I thought there are particular epistemic states that _don’t have words yet_, but should. And that a function of LessWrong might be to make various possible epistemic states more salient as options. You might have reacts for “approve/disapprove” and “agree/disagree”... but you might also want reactions that let you quickly and effortless express “this isn’t exactly false or bad but it’s subtly making this discussion worse.”

Fictionalized, Paraphrased Critch said “hmm, this reminds me of some particular epistemic states I recently noticed that don’t have names.”

“Go on”, said I.

“So, you know the feeling of being uncertain? And how it feels different to be 60% sure of something, vs 90%?”

“Sure.”

“Okay. So here’s two other states you might be in:

*   **75% sure that you’ll eventually be 99% sure,**
*   **80% sure that you’ll eventually be 90% sure.**

He let me process those numbers for a moment.

...

Then he continued: "Okay, now imagine you’re thinking about a particular AI system you’re designing, which might or might not be alignable.

“If you’re feeling 75% sure that you’ll eventually be 99% sure that that AI is safe, this means you think that _eventually_ you’ll have a clear understanding of the AI, such that you feel confident turning it on without destroying humanity. Moreover you expect to be able to convince _other people_ that it’s safe to turn it on without destroying humanity.

“Whereas if you’re 80% sure that eventually you’ll be 90% sure that it’ll be safe, _even in the future state_ where you’re better informed and more optimistic, you might still not actually be confident enough to turn it on. And even if for some reason you are, other people might disagree about whether you should turn it on.

“I’ve noticed people tracking how certain they are of something, without paying attention to whether their uncertainty is _possible to resolve_. And this has important ramifications for what kind of plans they can make. Some plans _require near-certainty._ Especially many plans that require group coordination.

“Makes sense”, said I. "Can I write this up as a blogpost?"

* * *

I’m not quite sure about the best name here, but this seems like a useful concept to have a handle for. Something like “unresolvable uncertainty?”
# Modeling AGI Safety Frameworks with Causal Influence Diagrams

We have written a paper that represents various frameworks for designing safe AGI (e.g., RL with reward modeling, CIRL, debate, etc.) as Causal Influence Diagrams (CIDs), to help us compare frameworks and better understand the corresponding agent incentives.

We would love to get comments, especially on

*   Are the depicted frameworks represented accurately?
*   Is the CID representation helpful?
*   Frameworks we did not include that would be useful to model this way?

The paper's abstract:

Proposals for safe AGI systems are typically made at the level of frameworks, specifying how the components of the proposed system should be trained and interact with each other. In this paper, we model and compare the most promising AGI safety frameworks using causal influence diagrams. The diagrams show the optimization objective and causal assumptions of the framework. The unified representation permits easy comparison of frameworks and their assumptions. We hope that the diagrams will serve as an accessible and visual introduction to the main AGI safety frameworks.
# How does one get invited to the alignment forum?

At the moment there is a link on the Alignment Forum to apply for membership. You can apply by submitting papers, blog posts or comments. However, there is very little in the way of detail of what kind of work they expect they expect in order for you to have a reasonable chance of being invited. I'm not saying that there should be specific criteria as I'm in favour of the moderators using their judgement. However, more detail on what they are looking for would provide encouragement for people to work towards achieving the level of knowledge and good judgement that they expect for their members.
# Arbital Scrape V2

I've scraped [http://arbital.com](http://arbital.com/) as the site is unusably slow and hard to search for me.

[https://drive.google.com/open?id=1b7dKhOzfMpFwngAeI8efeOzv147Lv_mx](https://drive.google.com/open?id=1b7dKhOzfMpFwngAeI8efeOzv147Lv_mx)

[https://mega.nz/#!SUQg1YxS!G7bmpNipcl1Ztugvr6l6sUJUu1D0gDg1-jA6oXLNEWg](https://mega.nz/#!SUQg1YxS!G7bmpNipcl1Ztugvr6l6sUJUu1D0gDg1-jA6oXLNEWg)

[https://emma-borhanian.github.io/arbital-scrape/](https://emma-borhanian.github.io/arbital-scrape/)

The scrape is locally browsable and plain HTML save for MathJax and a few interactive demos. Source code included (with git history).

![](https://i.imgur.com/fcxSMSf.png)

(previously [Arbital Scrape](https://www.lesswrong.com/posts/muKEBrHhETwN6vp8J/arbital-scrape))

Updates: Included source code, MathJax and link formatting, cross-linking, missing pages, etc

Source: [https://github.com/emma-borhanian/arbital-scrape](https://github.com/emma-borhanian/arbital-scrape)

Mirror: [www.obormot.net/arbital](https://www.obormot.net/arbital)
# Research Agenda in reverse: what *would* a solution look like?

I constructed my [AI alignment research agenda](https://www.lesswrong.com/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into) piece by piece, stumbling around in the dark and going down many false and true avenues.

But now it is increasingly starting to feel natural to me, and indeed, somewhat inevitable.

What do I mean with that? Well, let's look at the problem in reverse. Suppose we had an AI that was aligned with human values/preferences. How would you expect that to have been developed? I see four natural paths:

1. Effective proxy methods. For example, Paul's amplification and distillation, or variants of revealed preferences, or a similar approach. The point of this that it reaches alignment without defining what a preference fundamentally is; instead it uses some proxy for the preference to do the job.
2. Corrigibility: the AI is safe and corrigible, and along with active human guidance, manages to reach a tolerable outcome.
3. Something new: a bold new method that works, for reasons we haven't thought of today (this includes most strains of moral realism).
4. An actual grounded definition of human preferences.

So, if we focus on scenario 4, we need a few things. We need a fundamental definition of what a human preference is (since we know [this can't be defined purely from behaviour](https://arxiv.org/abs/1712.05812)). We need a method of combining contradictory and underdefined human preferences. We also need a method for taking into account human meta-preferences. And both these methods has to actually reach an output, and not get caught in loops.

If those are the requirements, then it's obvious why we need most of the elements of my research agenda, or something similar. We don't need the exact methods sketched out there, there may be other way of synthesising preferences and meta-preferences together. But the overall structure - a way of defining preferences, and ways of combining them that produce an output - seems, in retrospect, inevitable. The rest is, to some extent, just implementation details.
# What does the word "collaborative" mean in the phrase "collaborative truthseeking"?

I keep hearing this phrase, "collaborative truthseeking." Question: what kind of epistemic work is the word "collaborative" doing?

Like, when you (respectively I) say a thing and I (respectively you) hear it, that's going to result in some kind of state change in my (respectively your) brain. If that state change results in me (respectively you) making better [predictions](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) than I (respectively you) would have in the absence of the speech, then that's evidence for the hypothesis that at least one of us is "truthseeking."

But what's this "collaborative" thing about? How do speech-induced state changes result in better predictions if the speaker and listener are "collaborative" with each other? Are there any circumstances in which the speaker and listener being "collaborative" might result in _worse_ predictions?
# What are principled ways for penalising complexity in practice?

  

Previously I [asked about](https://www.lesswrong.com/posts/Q9hDFkvCSwi6cwPGy/how-is-solomonoff-induction-calculated-in-practice-1) Solomonoff induction but essentially I asked the wrong question. Richard_Kennaway pointed me in the direction of an answer to the question which I should have asked but after investigating I still had questions.

So:

If one has 2 possible models to fit to a data set, by how much should one penalise the model which has an additional free parameter?

A couple of options which I came across were:

[AIC](https://en.wikipedia.org/wiki/Akaike_information_criterion), which has a flat facter of e penalty for each additional parameter.

[BIC](https://en.wikipedia.org/wiki/Bayesian_information_criterion), which has a factor of √n penalty for each additional parameter.

where n is the number of data points.

On the one hand having a penalty which increases with n makes sense - a useful additional parameter should be able to provide more evidence the more data you have. On the other hand, having a penalty which increases with n means your prior will be different depending on the number of data points which seems wrong.

So, count me confused. Maybe there are other options which are more helpful. I don't know if the answer is too complex for a blog post but, if so, any suggestions of good text books on the subject would be great.

EDIT: johnswentworth has written a [sequence](https://www.lesswrong.com/s/onCRFFN7rGXTg3jyc) which expands on the answer which he gives below.
# Is it good practice to write questions/comments on old posts you're trying to understand? 

I've recently started working through AI safety posts written on LessWrong 1-3 years ago; in doing so I occasionally have questions/comments about the material. Is it considered good practice/in line with LW norms to write these as comments on the original, old posts? One hand I can see why "necro-ing" old posts would be frowned on, but I'm not sure where else to bring it up. You can look at my comment history for examples of what I mean (before I realized it might not be a good idea)
# False assumptions and leaky abstractions in machine learning and AI safety

*   The problems of e[mbedded agency](https://arxiv.org/pdf/1902.09469.pdf) are due to the notion of agency implicit in reinforcement learning being a leaky abstraction.
*   Machine learning problem statements often makes assumptions that are known to be false, for example, assuming i.i.d. data.
*   **Examining failure modes that result from false assumptions** and leaky abstractions is important for safety, (at least) because they create additional possibilities for [convergent rationality](https://www.lesswrong.com/posts/pLZ3bdeng4u5W8Yft/let-s-talk-about-convergent-rationality-1).
*   **Attempting to enforce the assumptions** implicit in machine learning problem statements is another important topic for safety research, since we do not fully understand the failure modes.
*   In practice, most machine learning research is done in settings where unrealistic assumptions are trivially enforced to a sufficiently high extent that it is reasonable to assume they are not violated (e.g. by the use of a fixed train/valid/test set, generated via pseudo-random uniform sampling from a fixed dataset).
*   We can (and probably should) do machine learning research that targets failure modes of common assumptions and methods of enforcing assumptions by (instead) creating settings in which these assumptions have the potential to be violated.
# GreaterWrong Arbital Viewer

You can now view Arbital through GreaterWrong: [https://arbital.greaterwrong.com/](https://arbital.greaterwrong.com/)

Some of Arbital's features are supported and some aren't; let me know in the comments if there's anything you're particularly missing.

Thanks to [emmab](https://www.lesswrong.com/posts/8pSh54GoJJqfHdndk/arbital-scrape-v2) for downloading the content.
# Systems Engineering Advancement Research Initiative

This is a linkpost for a research initiative at MIT I just discovered while following up on some earlier reading. I have linked to the Publications page to make it easiest for people to get in and start perusing.

The goal of this initiative is to improve state-of-the-art systems engineering, and in particular to be able to account for uncertainty and changing contexts during and after the design phase.

What drove me to bring it to the attention of this community is the MATE program, which stands for [Multi-Attribute Tradespace Exploration](http://seari.mit.edu/mate.php). This is interesting because it consists of defining desirable qualities and then [building a utility function](http://seari.mit.edu/documents/courses/PI27s/SEAri_SC-2010-PI27s-03-1.pdf) out of them, upon which design decisions will be based.

*   Among the publications are a series of applied attempts at building actual utility functions for real things, then using decision theory with them.
*   They increase the amount of things you can trade off for by expanding what they call the "Ilities" (reliability, versatility, etc). Among these new ilities is a more advanced notion of safety.
*   This means there is a body of applied work which has a concept of safety, calculated into utility functions, and operated on according to decision theory.

Further updates will be made in the comments as I finish readings.
# What's the best explanation of intellectual generativity?

Lately I've found myself wanting to make the argument that intellectual generativity is very important, and that you should be very careful with subtle forces that can corrode it.

"Generativity" is the sort of word that seems to come up a lot in casual conversations in my current circle but I just went looking for a good explanatory post and couldn't find one. I'm fairly confident that someone somewhere has talked about it (not necessarily on LW).

Curious if anyone knows of good existing writing?

And if anyone wanted to write up a fresh explanation that'd be cool as well. (A possible outcome is treating the answer section here as an opportunity to write a first draft that maybe turns into a post if there's consensus the answer is good)