# Rationality Reading Group: Introduction and A: Predictably Wrong

_This is part of a semi-monthly reading group on_ _Eliezer Yudkowsky's ebook, [Rationality: From AI to Zombies](https://intelligence.org/rationality-ai-zombies/). For more information about the group, see the [announcement post](/lw/lx0/rationality_from_ai_to_zombies_online_reading/)._

* * *

Welcome to the _Rationality_ reading group. This week we discuss the _Preface_ by primary author Eliezer Yudkowsky, _Introduction_ by editor & co-author Rob Bensinger, and the first sequence: _Predictably Wrong_. This sequence introduces the methods of rationality, including its two major applications: the search for truth and the art of winning. The desire to seek truth is motivated, and a few obstacles to seeking truth--systematic errors, or _biases_--are discussed in detail.  
  
This post summarizes each article of the sequence, linking to the original LessWrong posting where available, and offers a few relevant notes, thoughts, and ideas for further investigation. My own thoughts and questions for discussion are in the comments.  
  
Reading: _Preface_, _Biases: An Introduction_, and _Sequence A: Predictably Wrong_ (pi-xxxv and p1-42)  

* * *

Introduction
------------

**Preface**. Introduction to the ebook compilation by Eliezer Yudkowsky. Retrospectively identifies mistakes of the text as originally presented. Some have been corrected in the ebook, others stand as-is. Most notably the book focuses too much on belief, and too little on practical actions, especially with respect to our everyday lives. Establishes that the goal of the project is to teach _rationality_, those ways of thinking which are common among practicing scientists and the foundation of the Enlightenment, yet not systematically organized or taught in schools (yet).

**Biases: An Introduction**. Editor & co-author Rob Bensinger motivates the subject of rationality by explaining the dangers of systematic errors caused by \*cognitive biases\*, which the arts of rationality are intended to de-bias. Rationality is not about Spock-like stoicism -- it is about simply "doing the best you can with what you've got." The System 1 / System 2 dual process dichotomy is explained: if our errors are systematic and predictable, then we can instil behaviors and habits to correct them. A number of exemplar biases are presented. However a warning: it is difficult to recognize biases in your own thinking even after learning of them, and knowing about a bias may grant unjustified overconfidence that you yourself do not fall pray to such mistakes in _your_ thinking. To develop as a rationalist actual experience is required, not just learned expertise / knowledge. Ends with an introduction of the editor and an overview of the organization of the book.

### A. Predictably Wrong

**1\. [What do I mean by "rationality"?](/lw/31/what_do_we_mean_by_rationality/)** Rationality is a systematic means of forming true beliefs and making winning decisions. Probability theory is the set of laws underlying rational belief, "epistemic rationality": it describes how to process evidence and observations to revise ("update") one's beliefs. Decision theory is the set of laws underlying rational action, "instrumental rationality", independent of what one's goals and available options are. (p7-11)

**2\. [Feeling rational.](/lw/hp/feeling_rational/)** Becoming more rational can diminish feelings or intensify them. If one cares about the state of the world, it is expected that he or she should have an emotional response to the acquisition of truth. "That which can be destroyed by the truth should be," but also "that which the truth nourishes should thrive." The commonly perceived dichotomy between emotions and "rationality" \[sic\] is more often about fast perceptual judgements (System 1, emotional) vs slow deliberative judgements (System 2, "rational" \[sic\]). But both systems can serve the goal of truth, or defeat it, depending on how they are used. (p12-14)

**3\. [Why truth? and...](/lw/go/why_truth_and/)** Why seek the truth? Curiosity: to satisfy an emotional need to know. Pragmatism: to accomplish some specific real-world goal. Morality: to be virtuous, or fulfill a duty to truth. Curiosity motivates a search for the most intriguing truths, pragmatism the most useful, and morality the most important. But be wary of the moral justification: "To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom. People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake." (p15-18)  
  
**4\. [...what's a bias, again?](/lw/gp/whats_a_bias_again/)** A bias is an obstacle to truth, specifically those obstacles which are produced by our own thinking processes. We describe biases as failure modes which systematically prevent typical human beings from determining truth or selecting actions that would have best achieved their goals. Biases are distinguished from mistakes which originate from false beliefs or brain injury. Do better seek truth and achieve our goals we must identify our biases and do what we can to correct for or eliminate them. (p19-22)

**5\. [Availability.](/lw/j5/availability/)** The availability heuristic is judging the frequency or probability of an event by the ease with which examples of the event come to mind. If you think you've heard about murders twice as much as suicides then you might suppose that murder is twice as common as suicide, when in fact the opposite is true. Use of the availability heuristic gives rise to the absurdity bias: events that have never happened are not recalled, and hence deemed to have no probability of occurring. In general, memory is not always a good guide to probabilities in the past, let alone to the future. (p23-25)

**6\. [Burdensome details.](/lw/jk/burdensome_details/)** The conjunction fallacy is when humans rate the probability of two events has higher than the probability of either event alone: adding detail can make a scenario _sound more plausible_, even though the event as described necessarily _becomes less probable_. Possible fixes include training yourself to notice the addition of details and discount appropriately, thinking about other reasons why the central idea could be true other than the added detail, or training oneself to hold a preference for simpler explanations -- to feel _every_ added detail as a burden. (p26-29)

**7\. [Planning fallacy.](/lw/jg/planning_fallacy/)** The planning fallacy is the mistaken belief that human beings are capable of making accurate plans. The source of the error is that we tend to imagine how things will turn out if everything goes according to plan, and do not appropriately account for possible troubles or difficulties along the way. The typically adequate solution is to compare the new project to broadly similar previous projects undertaken in the past, and ask how long those took to complete. (p30-33)

**8\. [Illusion of transparency: why no one understands you.](/lw/ke/illusion_of_transparency_why_no_one_understands/)** The illusion of transparency is our bias to assume that others will understand the intent behind our attempts to communicate. The source of the error is that we do not sufficiently consider alternative frames of mind or personal histories, which might lead the recipient to alternative interpretations. Be not too quick to blame those who misunderstand your perfectly clear sentences, spoken or written. Chances are, your words are more ambiguous than you think. (p34-36)

**9\. [Expecting short inferential distances.](/lw/kg/expecting_short_inferential_distances/)** Human beings are generally capable of processing only one piece of new information at at time. Worse, someone who says something with no obvious support is a liar or an idiot, and if you say something blatantly obvious and the other person doesn't see it, _they're_ the idiot. This is our bias towards explanations of short inferential distance. A clear argument has to lay out an inferential _pathway_, starting from what the audience already knows or accepts. If at any point you make a statement without obvious justification in arguments you've previously supported, the audience just thinks you're crazy. (p37-39)

**10\. [The lens that sees its own flaws.](/lw/jm/the_lens_that_sees_its_flaws/)** We humans have the ability to introspect our own thinking processes, a seemingly unique skill among life on Earth. As consequence, a human brain is able to understand its own flaws--its systematic errors, its biases--and apply second-order corrections to them. (p40-42)

* * *

It is at this point that I would generally like to present an opposing viewpoint. However I must say that this first introductory sequence is not very controversial! Educational, yes, but not controversial. If anyone can provide a link or citation to one or more decent non-strawman arguments which oppose any of the ideas of this introduction and first sequence, please do so in the comments. I certainly encourage awarding karma to anyone that can do a reasonable job [steel-manning](http://wiki.lesswrong.com/wiki/Steel_man) an opposing viewpoint.

* * *

This has been a collection of notes on the assigned sequence for this week. **The most important part of the reading group though is discussion, which is in the comments section.** I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover _Sequence B: Fake Beliefs_ (p43-77). The discussion will go live on **Wednesday, 6 May 2015** at or around **6pm PDT**, right here on the discussion forum of LessWrong.