Harley-ml/lesswrong huggingface dataset:
# Loebian cooperation, version 2

I'm writing up some math results developed on LW as a paper with the tentative title "Self-referential decision algorithms". Something interesting came up while I was cleaning up the [Loebian cooperation](/lw/2ip/ai_cooperation_in_practice) result. Namely, how do we say precisely that Loebian cooperation is stable under minor syntactic changes? After all, if we define a "minor change" to program A as a change that preserves A's behavior against any program B, then [quining cooperation](/lw/do/reformalizing_pd/) is just as stable under such "minor changes" by definition. Digging down this rabbit hole, I seem to have found a nice new reformulation of the whole thing.

I will post some sections of my current draft in the comments to this post. Eventually this material is meant to become an academic paper (hopefully), so any comments on math mistakes, notation or tone would be much appreciated! And yeah, I have no clue about academic writing, so you're welcome to tell me that too.
# Proposal for "Open Problems in Friendly AI"

Series: [How to Purchase AI Risk Reduction](/r/discussion/lw/cs6/how_to_purchase_ai_risk_reduction/)

[![](http://commonsenseatheism.com/wp-content/uploads/2012/05/Open-FAI-preview-screenshot.png)](http://commonsenseatheism.com/wp-content/uploads/2012/05/Open-Problems-in-Friendly-Artificial-Intelligence.pdf)[One](/lw/cnj/a_scholarly_ai_risk_wiki/) [more](/lw/cok/funding_good_research/) [project](/lw/cr0/short_primers_on_crucial_topics/) SI is considering...

When I was hired as an intern for SI in April 2011, one of my first proposals was that SI create a technical document called _Open Problems in Friendly Artificial Intelligence_. ([Here](http://commonsenseatheism.com/wp-content/uploads/2012/05/Open-Problems-in-Friendly-Artificial-Intelligence.pdf) is a preview of what the document would be like.)

When someone becomes persuaded that Friendly AI is important, their first question is often: "Okay, so what's the technical research agenda?"

[So You Want to Save the World](http://lukeprog.com/SaveTheWorld.html) maps out some broad _categories of research questions_, but it doesn't explain what the _technical research agenda_ is. In fact, SI hasn't yet explained much of the technical research agenda yet.

_Much_ of the technical research agenda should be kept secret for the same reasons you might want to keep secret the DNA for a synthesized supervirus. But _some_ of the Friendly AI technical research agenda is safe to explain so that a broad research community can contribute to it.

This research agenda includes:

*   Second-order logical version of Solomonoff induction.
*   Non-Cartesian version of Solomonoff induction.
*   Construing utility functions from psychologically realistic models of human decision processes.
*   Formalizations of value extrapolation. (Like [Christiano's attempt](http://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/).)
*   Microeconomic models of self-improving systems (e.g. takeoff speeds).
*   ...and several others open problems.

The goal would be to define the open problems as formally and precisely as possible. Some will be more formalizable than others, at this stage. (As a model for this kind of document, see Marcus Hutter's [Open Problems in Universal Induction and Intelligence](http://www.hutter1.net/ai/aixiopen.pdf).)

Nobody knows the open problems in Friendly AI research better than [Eliezer](http://yudkowsky.net/), so it would probably be best to approach the project this way:

1.  Eliezer spends a month writing an "Open Problems in Friendly AI" sequence for Less Wrong.
2.  Luke organizes a (fairly large) research team for presenting these open problems with greater clarity and thoroughness, in the mainstream academic form.
3.  These researchers collaborate for several months to put together the document, involving Eliezer when necessary.
4.  SI publishes the final document, possibly in a journal.

Estimated cost:

*   2 months of Eliezer's time.
*   150 hours of Luke's time.
*   $40,000 for contributed hours from staff researchers, remote researchers, and perhaps domain experts (as consultants) from mainstream academia.
# How to Purchase AI Risk Reduction

I'm writing a series of discussion posts on **how to purchase AI risk reduction** (through [donations to the Singularity Institute](http://intelligence.org/donate/), anyway; other x-risk organizations will have to speak for themselves about their plans).

Each post outlines a concrete proposal, with cost estimates:

*   [A scholarly AI risk wiki](/lw/cnj/a_scholarly_ai_risk_wiki/)
*   [Funding good research](/r/discussion/lw/cok/funding_good_research/)
*   [Short primers on crucial topics](/lw/cr0/short_primers_on_crucial_topics/)
*   ["Open Problems in Friendly AI"](/lw/cr7/proposal_for_open_problems_in_friendly_ai/)
*   [Building the AI risk research community](/r/discussion/lw/cr6/building_the_ai_risk_research_community/)
*   [Reaching young math/compsci talent](/r/discussion/lw/cs7/reaching_young_mathcompsci_talent/)
*   [Raising safety-consciousness among AGI researchers](/r/discussion/lw/cu6/raising_safetyconsciousness_among_agi_researchers/)
*   [Strategic research on AI risk](/r/discussion/lw/cua/strategic_research_on_ai_risk/)
*   [Building toward a Friendly AI team](/r/discussion/lw/cv9/building_toward_a_friendly_ai_team/)

Also see John Maxwell's [Brainstorming additional AI risk reduction ideas](/lw/d34/brainstorming_additional_ideas_on_how_to_purchase/).

(For a quick primer on AI risk, see _[Facing the Singularity](http://facingthesingularity.com/)_.)

[![](http://commonsenseatheism.com/wp-content/uploads/2012/05/FtS-medium.png)](http://facingthesingularity.com/)
# Son of Shit Rationalists Say

A long time ago, in the colder seasons, I [asked for suggestions](/lw/9ki/shit_rationalists_say/) for a Shit Rationalists Say video. Due to other concerns it took me this long to put it together, and the meme has long since passed. However, here it is.

[Shit Rationalists Say](http://www.youtube.com/watch?v=jlT3MeCzVao)

It is my first time in front of a camera, so I'm shakey. But I learned, and there it is.
# Marketplace Transactions Open Thread

Social scientists think humans operate under one of two different sets of norms, depending on the circumstances: "market norms" or "social norms". The basic idea is that when exchanging money for goods and services formally, it's considered okay to be much more calculating and self-interested than when exchanging favors with friends informally. You can read [this blog post](http://danariely.com/the-books/excerpted-from-chapter-4-–-the-cost-of-social-norms/) by Dan Ariely for more.

It's often considered rude to introduce market norms in an area where they don't traditionally apply. For example, by [charging money for your presence at a barbecue](http://www.reddit.com/r/todayilearned/comments/ufl15/til_carl_sagan_sued_apple_for_using_his_name_as/c4v03on).

This is a thread where it's okay to talk about trading money for goods and services with other Less Wrong users, which might otherwise be considered rude because you'd be inappropriately introducing market norms. Things you're encouraged to do include:

*   Post your resume
*   Advertise a product sold by you or your company
*   Advertise a service provided by you or your company
*   Advertise an open position working for you or your company

The argument for having a thread like this is as follows. Less Wrong users have a variety of goals they wish to accomplish. Some of these goals involve engaging in marketplace transactions. It's plausible that a thread facilitating marketplace transactions between LW users will buy just as much or more collective goal accomplishment per unit attention consumed than a traditional Less Wrong thread.

Anecdotally it seems that introducing market norms takes a certain amount of chutzpah. For example, apparently it takes a certain kind of person to actually be able to name a dollar figure in a sales conversation, and that's why you need a professional salesperson to come along with a sales engineer when selling a technical product. One LWer friend of mine struggled for a while before she was able to get herself to charge money for talk therapy she had been providing to friends for free.

To combat this, please feel inclined to vote up folks who post in this thread. They likely overcame some akrasia in the act of promoting their offer.

To discuss the concept of this thread, as opposed to advertising a transaction you wish to engage in, please reply to [this comment](/r/discussion/lw/ctg/marketplace_transactions_open_thread/6q87).
# Paper: Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent

Bill "Numerical Recipes" Press and Freeman "Dyson sphere" Dyson [have a new paper on iterated prisoner dilemas](http://www.pnas.org/content/early/2012/05/16/1206569109.full.pdf) (IPD). Interestingly they found new surprising results:

> It is generally assumed that there exists no simple ultimatum strategy whereby one player can enforce a unilateral claim to an unfair share of rewards. Here, we show that such strategies unexpectedly do exist. In particular, a player X who is witting of these strategies can (i) deterministically set her opponent Y’s score, independently of his strategy or response, or (ii) enforce an extortionate linear relation between her and his scores.

They discuss a special class of strategies - zero determinant (ZD) strategies of which tit-for-tat (TFT) is a special case:

> The extortionate ZD strategies have the peculiar property of sharply distinguishing between “sentient” players, who have a theory of mind about their opponents, and “evolutionary” players, who may be arbitrarily good at exploring a ﬁtness landscape (either locally or globally), but who have no theory of mind.

The evolutionary player adjusts his strategy to maximize score, but doesn't take his opponent explicitly into account in another way (hence has "no theory of mind" of the opponent). Possible outcomes are:

A)

> If X alone is witting of ZD strategies, then IPD reduces to one of two cases, depending on whether Y has a theory of mind. If Y has a theory of mind, then IPD is simply an ultimatum game (15, 16), where X proposes an unfair division and Y can either accept or reject the proposal. If he does not (or if, equivalently, X has ﬁxed her strategy and then gone to lunch), then the game is dilemma-free for Y. He can maximize his own score only by giving X even more; there is no beneﬁt to him in defecting.

B)

> If X and Y are both witting of ZD, then they may choose to negotiate to each set the other’s score to the maximum cooperative value. Unlike naive PD, there is no advantage in defection, because neither can affect his or her own score and each can punish any irrational defection by the other. Nor is this equivalent to the classical TFT strategy (7), which produces indeterminate scores if played by both players.

This latter case sounds like a formalization of Hosfstadter's superrational agents. The cooperation enforcement via cross-setting the scores is very interesting.

Is this connection true or am I misinterpreting it? (This is not my field and I've only skimmed the paper up to now.) What are the implications for FAI? If we'd get into an IPD situation with an agent for which we simply can not put together a theory of mind, do we have to live with extortion? What would effectively mean to have a useful theory of mind in this case?

The paper ends in a grand style (spoiler alert):

> It is worth contemplating that, though an evolutionary player Y is so easily beaten within the conﬁnes of the IPD game, it is exactly evolution, on the hugely larger canvas of DNA-based life, that ultimately has produced X, the player with the mind.
# Raising safety-consciousness among AGI researchers

Series: [How to Purchase AI Risk Reduction](/r/discussion/lw/cs6/how_to_purchase_ai_risk_reduction/)

Another method for purchasing AI risk reduction is to raise the safety-consciousness of researchers doing work related to AGI.

The Singularity Institute is conducting a study of scientists who decided to either (1) stop researching some topic after realizing it might be dangerous, or who (2) forked their career into advocacy, activism, ethics, etc. because they became concerned about the potential negative consequences of their work. From this historical inquiry we hope to learn some things about what causes scientists to become so concerned about the consequences of their work that they _take action_. Some of the examples we've found so far: Michael Michaud (resigned from SETI in part due to worries about the safety of trying to contact ET), Joseph Rotblat (resigned from the Manhattan Project before the end of the war due to concerns about the destructive impact of nuclear weapons), and Paul Berg (became part of a self-imposed moratorium on recombinant DNA back when it was still unknown how dangerous this new technology could be).

What else can be done?

*   Academic outreach, in the form of [conversations with AGI researchers](http://wiki.lesswrong.com/wiki/Muehlhauser_interview_series_on_AGI) and "basics" papers like [Intelligence Explosion: Evidence and Import](http://commonsenseatheism.com/wp-content/uploads/2012/02/Muehlhauser-Salamon-Intelligence-Explosion-Evidence-and-Import.pdf) or [Complex Value Systems are Required to Realize Valuable Futures](http://intelligence.org/upload/complex-value-systems.pdf).
*   [A scholarly AI risk wiki](/lw/cnj/a_scholarly_ai_risk_wiki/).
*   [Short primers on crucial topics](/lw/cr0/short_primers_on_crucial_topics/).
*   Whatever is suggested by our analysis of past researchers who took action in response to their concerns about the ethics of their research, and by other analyses of human behavior.

Naturally, these efforts should be directed toward researchers who are both highly competent and whose work is very relevant to development toward AGI: researchers like [Josh Tenenbaum](http://web.mit.edu/cocosci/josh.html), [Shane Legg](http://www.vetta.org/), and [Henry Markram](http://en.wikipedia.org/wiki/Henry_Markram).
# "Where Am I?", by Daniel Dennett 

[”Where Am I?”](http://www.newbanner.com/SecHumSCM/WhereAmI.html) is a short story by [Daniel Dennett](http://en.wikipedia.org/wiki/Daniel_Dennett) from his book _Brainstorms: Philosophical Essays on Mind and Psychology._Some of you might already be familiar with it.

The story is a humorous semi-science fiction one, where Dennett gets a job offer form Pentagon that entails moving his brain into a vat, without actually moving his point of view. Later on it brings up questions about uploading and what it would mean in terms of diverging perspectives and so on. Aside from being a joy to read, it offers solutions to a few hurdles about the _nature of consciousnesses _and _personal identity. _  
  

> Suppose, I argued to myself, I were now to fly to California, rob a bank, and be apprehended.  In which state would I be tried:  in California, where the robbery took place, or in Texas, where the brains of the outfit were located?  Would I be a California felon with an out-of-state brain, or a Texas felon remotely controlling an accomplice of sorts in California? It seemed possible that I might beat such a rap just on the undecidability of that jurisdictional question, though perhaps it would be deemed an interstate, and hence Federal, offense.
# Satire of Journal of Personality and Social Psychology's publication bias

Follow-up to: [Follow-up on ESP study: "We don't publish replications"](/lw/6lq/followup_on_esp_study_we_dont_publish_replications/), [Using degrees of freedom to change the past for fun and profit](/lw/a9f/using_degrees_of_freedom_to_change_the_past_for/)

As I discussed in the above posts, the Journal of Personality and Social Psychology, a leading psych journal, published a deeply flawed parapsychology study (see the second post for details) which had apparently been tortured to produce results. Then they rejected an attempt to replicate that found no effect, citing a sadly typical policy of not publishing replications. Some of you may enjoy reading one enterprising researcher's [amusing satire article](http://www.projectimplicit.net/arina/B2012.pdf), purportedly (not actually) "tallying" past confirmations and disconfirmations in JPSP and drawing conclusions.

ETA: To clarify the last sentence, they didn't really find 4800+ confirmation and two disconfirmations. As they say in small print, the data were made up. It's right by the chart.
# Global Workspace Theory

Much research has been done upon visual perception.   Humans have the illusion that they are directly aware of everything in their 'field of view', but it turns out that they actually navigate not through reality, but through a model of reality that their brain stitches together mainly from the bits the eye is directly looking at as it darts about, with the rest supplied by interpolation based upon expectations.

For more info, read [The Illusion of Continuity: Active Perception and the Classical Editing System](http://sensation.vizkult.hu/wp-content/uploads/2012/05/Berliner.Cohen_.continuity.pdf), by Berliner and Cohen.

[Global Workspace Theory](http://en.wikipedia.org/wiki/Global_Workspace_Theory) is the idea that our awareness of our own thought process works the same way.  We have the illusion of an unbroken stream of consciousness, but what we're actually referencing is a model of what the brain thinks it has been consciously thinking about, that is stitched together from brief fragments, the way a spotlight in a theatre might move about shining on different parts of a stage, revealing actors making speeches and interacting with each other.   Even when the spotlight moves on, the actors, stage hands and directors remain and keep working.   When the spotlight returns, to catch a later part of the drama in that area, we interpolate what the actors would have been doing while we were paying attention elsewhere.
# List of Problems That Motivated UDT

I noticed that recently I wrote several comments of the form "UDT can be seen as a step towards solving X" and thought it might be a good idea to list in one place all of the problems that helped motivate [UDT1](/lw/15m/towards_a_new_decision_theory/) (not including problems that came up subsequent to that post). 

*   [decision making for minds that can copy themselves](http://extropians.weidai.com/extropians.3Q97/4116.html)
*   [Doomsday Argument](http://www.weidai.com/everything.html)
*   Sleeping Beauty
*   [Absent-Minded Driver](http://wiki.lesswrong.com/wiki/Absent-Minded_driver)
*   [Presumptuous Philosopher](/lw/175/torture_vs_dust_vs_the_presumptuous_philosopher/)
*   [anthropic reasoning for non-sentient AIs](http://groups.google.com/group/everything-list/browse_thread/thread/8c25168e232a7efd/)
*   [Simulation Argument](/lw/1lq/less_wrong_qa_with_eliezer_yudkowsky_video_answers/1f0b)
*   [indexical uncertainty](/lw/102/indexical_uncertainty_and_the_axiom_of/) in general
*   [wireheading/Cartesianism](http://www.mail-archive.com/everything-list@googlegroups.com/msg03620.html) ([how](/lw/cej/general_purpose_intelligence_arguing_the/6mm0) to formulate something like AIXI that cares about an external world instead of just its sensory inputs)
*   [How](http://extropians.weidai.com/extropians/0302/2444.html) to make decisions if all possible worlds exist? (a la Tegmark or Schmidhuber, or just in the MWI)
*   Quantum Immortality/Suicide
*   Logical Uncertainty ([how](http://www.sl4.org/archive/0509/12317.html) to formulate something like Godel machine that can make reasonable decisions involving P=NP)
*   [uncertainty about hypercomputation](https://groups.google.com/group/everything-list/browse_thread/thread/c7442c13ff1396ec) (how to avoid assuming we must be living in a computable universe)
*   [What are probabilities?](/lw/1iy/what_are_probabilities_anyway/)
*   [What are decisions](/lw/aq9/decision_theories_a_less_wrong_primer/606k) and what kind of consequences should be considered when making decisions?
*   Newcomb's Problem
*   [Smoking Lesion](/lw/az7/video_paul_christianos_impromptu_tutorial_on_aixi/6cv7)
*   Prisoner's Dilemma
*   [Counterfactual Mugging](http://wiki.lesswrong.com/wiki/Counterfactual_mugging)
*   [FAI](http://extropians.weidai.com/extropians/0302/2567.html)
# Help please!

Yesterday my mom noticed (at a funeral) that I wasn't praying or participating in the mass. She confronted me about it, and I told her that no, I am not Catholic. Apparently it's sinking in and she's a bit hysterical... crying and screaming that she doesn't know me anymore.

What do I do? I don't know how to react/behave when she's doing this. It's like she wants me to feel like I'm doing something wrong, but it isn't working, so she's getting hysterical.

\*edit\*

I gave her a hug when she calmed down and told her I love her. That seemed to help, a little. Based on her previous behavior in situations where I've done something "wrong," she will (in the future) make barbs and slight passes at my beliefs. (Already she made one: insisting my love of science is causing my social anxiety disorder.) The advice given in the comments is really helpful. I plan on making the most of it.
# Peter Thiel's AGI discussion in his startups class @ Stanford [link]

[http://blakemasters.tumblr.com/post/24464587112/peter-thiels-cs183-startup-class-17-deep-thought](http://blakemasters.tumblr.com/post/24464587112/peter-thiels-cs183-startup-class-17-deep-thought)

Some perspectives on AI risk that might be interesting. Peter is (the primary?) donor for SI, and an investor in AGI startup Vicarious Systems.
# Ask an experimental physicist

In response to falenas108's "Ask an X" thread. I have a PhD in experimental particle physics; I'm currently working as a postdoc at the University of Cincinnati. Ask me anything, as the saying goes.

This is an experiment. There's nothing I like better than talking about what I do; but I usually find that even quite well-informed people don't know enough to ask questions sufficiently specific that I can answer any better than the next guy. What goes through most people's heads when they hear "particle physics" is, judging by experience, string theory. Well, I dunno nuffin' about string theory - at least not any more than the average layman who has read Brian Greene's book. (Admittedly, neither do string theorists.) I'm equally ignorant about quantum gravity, dark energy, quantum computing, and the Higgs boson - in other words, the big theory stuff that shows up in popular-science articles. For that sort of thing you want a theorist, and not just any theorist at that, but one who works specifically on that problem. On the other hand I'm reasonably well informed about production, decay, and mixing of the charm quark and charmed mesons, but who has heard of that? (Well, now you have.) I know a little about CP violation, a bit about detectors, something about reconstructing and simulating events, a fair amount about how we extract signal from background, and quite a lot about fitting distributions in multiple dimensions.
# Combining causality with algorithmic information theory

Warning: maths.

[Causal inference using the algorithmic Markov condition](http://arxiv.org/abs/0804.3678) (Janzing and Schölkopf, 2008) replaces conditional independences between random variables, which define the structure of causal graphs, with algorithmic conditional independences between bit strings.

Conditional probabilities between variables become conditional complexities between strings, i.e. K(x|y) is the length of the shortest program that can generate the string x from y. Similarly, algorithmic mutual information I(x:y) is the amount of information that can be omitted in defining a string y given a shortest compressor for string x, I(x:y) = K(y) - K(y|x*). K(x,y) is the complexity of the concatenation of two strings x and y. These lead naturally to a definition of algorithmic conditional independence as I(x:y|z) = K(x|z) + K(y|z) - K(x,y|z) = 0 , where equality is defined up to the standard additive constant.

Then a lot of sexy, confusing proofs happen. When the dust settles, it looks like if you take some strings describing observations, interpret them as nodes in a graph, and "factor" so that a certain algorithmic Markov condition holds (every node string should be algorithmically independent of its non-descendant node strings given the optimal compressor of its parents' node strings), then every node can be computed by an O(1) program run on a Turing machine, with the node's parents and a noise term as input (with each node's noise string being jointly independent of the others). 

Notably, this means that if we make two observations which were "generated from their parents by the same complex rule", then we can "postulate another causal link between the nodes that explains the similarity of mechanisms". They say "complex rule" because the mutual algorithmic information between simple information strings, like some digits of pi, will be swallowed up by additive constants. Which all seems very close to rediscovering TDT.

There's more to the paper, but that's the tasty bit, so the summary ends here.
# What was your biggest recent surprise?

I recently flipped through the "Cartoon Guide to Physics", expecting an easy-to-understand rehash of ideas I was long familiar with; and that's what I got - right up to the last few pages, where I was presented with a fairly fundamental concept that's been absent from the popular science media I've enjoyed over the years. (Specifically, that the uncertainty principle, when expressed as linking energy and time, explains what electromagnetic fields actually /are/, as the propensity for virtual photons of various strengths to happen.) I find myself happy to try to integrate this new understanding - and at least mildly disturbed that I'd been missing it for so long, and with an increased curiosity about how I might find any other such gaps in my understanding of how the universe works.

So: what's the biggest, or most surprising, or most interesting concept /you/ have learned of, after you'd already gotten a handle on the basics?
# I think I've found the source of what's been bugging me about "Friendly AI"

In the comments on [this post](/lw/cjx/how_likely_the_ai_that_knows_its_evil_or_is_a/) (which in retrospect I feel was not very clearly written), someone linked me to a post Eliezer wrote five years ago, ["The Hidden Complexity of Wishes."](/lw/ld/the_hidden_complexity_of_wishes/) After reading it, I think I've figured out why the term ["Friendly AI" is used so inconsistently.](/lw/cib/work_harder_on_tabooing_friendly_ai/)

This post explicitly lays out a view that seems to be implicit in, but not entirely clear from, many of of Eliezer's other writings. That view is this:

> There are three kinds of genies:  Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which _no_ wish is safe; and [genies that aren't very powerful or intelligent](/lw/l8/conjuring_an_evolution_to_serve_you/).

Even if Eliezer is right about that, I think that view of his has led to confusing usage of the term "Friendly AI." If you accept Eliezer's view, it may seem to make sense to not worry to much about whether by "Friendly AI" you mean:

1.  A utopia-making machine (the AI "to whom you can safely say, 'I wish for you to do what I should wish for.'") Or:
    
2.  A non-doomsday machine (a doomsday machine being the AI "for which _no_ wish is safe.")
    

And it would make sense not to worry too much about that distinction, if you were talking only to people who also believe those two concepts are very nearly co-extensive for powerful AI. But failing to make that distinction is obviously going to be confusing when you're talking to people who don't think that. It will make it harder to communicate both your ideas and your reasons for holding those ideas to them.

One solution would be to more frequently link people back to "The Hidden Complexity of Wishes" (or other writing by Eliezer that makes similar points--what else would be suitable?) But while it's a good post and Eliezer makes some very good points with the "Outcome Pump" thought-experiment, the argument isn't entirely convincing.

As Eliezer himself has [argued at great length,](/lw/lp/fake_fake_utility_functions/) (see also section 6.1 of [this paper](/singinst.org/upload/artificial-intelligence-risk.pdf)) humans' own understanding of our values is far from perfect. None of us are, right now, qualified to design a utopia. But we do have some understanding of our own values; we can identify some things that would be improvements over our current situation while marking other scenarios as "this would be a disaster." It seems like there might be a point in the future where we can design an AI whose understanding of human values is similarly serviceable but no better than that.

Maybe I'm wrong about that. But if I am, until there's a better easy to read explanation of why I'm wrong for everybody to link to, it would be helpful to have different terms for (1) and (2) above. Perhaps call them "utopia AI" and "safe AI," respectively?
# [link] Aubrey de Grey answers Reddit AMA in video

[Aubrey de Grey](http://en.wikipedia.org/wiki/Aubrey_de_Grey "Aubrey de Grey"), Chief Science Officer of the SENS Foundation, has posted a video with answers to some of the questions posed at him in a recent [Reddit AMA](http://www.reddit.com/r/IAmA/comments/to32n/ask_aubrey_de_grey_chief_science_officer_sens/ "Reddit AMA").

Link: http://www.youtube.com/watch?v=6eet44YacRg
# [Link] Some notes on Rationality in Harry Potter and the Methods of Rationality

[http://www.etla.org/hpmor_notes.html](http://www.etla.org/hpmor_notes.html)

This is an ongoing project of mine, although I haven't worked on it in a while. I've been trying to extract the references to Rationality - the Methods of Rationality from [HPMoR](http://www.hpmor.com/). It also ended up having a few quotes that seemed interesting about how the story's going. I've linked references where I could find them.

I've only got as far as Chapter 40. Any extra submissions welcome.

At least one person - [User:DavidGerard](http://wiki.lesswrong.com/wiki/User:David_Gerard) suggested it deserved being posted as a discussion link.
# Let's all learn stats!

[I want to become Stronger!](/lw/h8/tsuyoku_naritai_i_want_to_become_stronger/)

[Udacity](http://www.udacity.com/) are running an [Introduction to Statistics](http://www.udacity.com/overview/Course/st101/CourseRev/1) course starting on the 25th June 2012.

Many of us could stand to learn some more stats, I certainly could. This seems like a great opportunity!

It is mandatory for all LWers to enroll in this course.

Update: the last line was a joke. Obviously people are not finding it funny. Sorry.
# How to Run a Successful Less Wrong Meetup

[![](http://commonsenseatheism.com/wp-content/uploads/2012/05/cover-of-meetup-booklet.png)](http://wiki.lesswrong.com/mediawiki/images/c/ca/How_to_Run_a_Successful_Less_Wrong_Meetup_Group.pdf)Always wanted to run a Less Wrong meetup, but been unsure of how? The _[How to Run a Successful Less Wrong Meetup](http://wiki.lesswrong.com/mediawiki/images/c/ca/How_to_Run_a_Successful_Less_Wrong_Meetup_Group.pdf)_ booklet is here to help you!

The 33-page document draws from consultations with more than a dozen Less Wrong meetup group organizers. Stanislaw Boboryk created the document design. Luke provided direction, feedback, and initial research, and I did almost all the writing.

The booklet starts by providing some motivational suggestions on _why_ you'd want to create a meetup in the first place, and then moves on to the subject of organizing your first one. Basics such as choosing a venue, making an announcement, and finding something to talk about once at the meetup, are all covered. This section also discusses pioneering meetups in foreign cities and restarting inactive meetup groups.

For those who have already established a meetup group, the booklet offers suggestions on things such as attracting new members, maintaining a pleasant atmosphere, and dealing with conflicts within the group. The "How to Build Your Team of Heroes" section explains the roles that are useful for a meetup group to fill, ranging from visionaries to organizers.

If you're unsure of what exactly to _do_ at meetups, the guide describes many options, from different types of discussions to nearly 20 different games and exercises. All the talk and philosophizing in the world won't do much good if you don't actually _do_ things, so the booklet also discusses long-term projects that you can undertake. Some people attend meetups to just have fun and to be social, and others to improve themselves and the world. The booklet has been written to be useful for both kinds of people.

In order to inspire you and let you see what others have done, the booklet also has brief case studies and examples from real meetup groups around the world. You can find these sprinkled throughout the guide.

**This is just the first version of the guide.** We _will_ continue working on it. If you find mistakes, or think that something is unclear, or would like to see some part expanded, or if you've got good advice you think should be included... please let me know! You can contact me at kaj.sotala@intelligence.org.

A large number of people have helped in various ways, and I hope that I've remembered to mention most of them in the acknowledgements. If you've contributed to the document but don't see your name mentioned, please send me a message and I'll have that fixed!

The booklet has been illustrated with pictures from various meetup groups. Meetup organizers sent me the pictures for this use, and I explicitly asked them to make sure that everyone in the photos was fine with it. Regardless, if there's a picture that you find objectionable, please contact me and I'll have it replaced with something else.
# [Link] Can We Reverse The Stanford Prison Experiment?

From the Harvard Business Review, an article entitled: "Can We Reverse The Stanford Prison Experiment?"

By: Greg McKeown  
Posted: June 12, 2012

[Clicky Link of Awesome! Wheee! Push me!](http://blogs.hbr.org/cs/2012/06/can_we_reverse_the_stanford_pr.html)

**Summary**:

Royal Canadian Mounted Police attempt a program where they hand out "Positive Tickets" 

> Their approach was to try to catch youth doing the right things and give them a [Positive Ticket](http://www.positivetickets.com/). The ticket granted the recipient free entry to the movies or to a local youth center. They gave out an average of 40,000 tickets per year. That is three times the number of negative tickets over the same period. As it turns out, and unbeknownst to Clapham, that ratio (2.9 positive affects to 1 negative affect, to be precise) is called the [Losada Line](http://en.wikipedia.org/wiki/Losada_line). It is the minimum ratio of positive to negatives that has to exist for a team to flourish. On higher-performing teams (and marriages for that matter) the ratio jumps to [5:1](http://en.wikipedia.org/wiki/Positivity/negativity_ratio). But does it hold true in policing?
> 
> According to Clapham, youth recidivism was reduced from 60% to 8%. Overall crime was reduced by 40%. Youth crime was cut in half. And it cost one-tenth of the traditional judicial system.

  
This idea can be applied to Real Life

> The lesson here is to create a culture that _immediately_ and sincerely celebrates victories. Here are three simple ways to begin:
> 
> **1\. Start your next staff meeting with five minutes on the question: "What has gone right since our last meeting?"** Have each person acknowledge someone else's achievement in a concrete, sincere way. Done right, this very small question can begin to shift the conversation.
> 
> **2\. Take two minutes every day to try to catch someone doing the right thing.** It is the fastest and most positive way for the people around you to learn when they are getting it right.
> 
> **3\. Create a virtual community board where employees, partners and even customers can share what they are grateful for daily. **Sounds idealistic? Vishen Lakhiani, CEO of [Mind Valley](http://www.mindvalley.com/get-published/), a new generation media and publishing company, has done just that at [Gratitude Log](http://www.gratitudelog.com/vishen). (Watch him explain how it works [here](http://www.youtube.com/watch?v=RKCb82ZzZyg)).
# Brainstorming additional AI risk reduction ideas

It looks as though lukeprog has finished his series on how to purchase AI risk reduction. But the ideas lukeprog shares are not the only available strategies. Can Less Wrong come up with more?

A summary of recommendations from [Exploring the Idea Space Efficiently](/lw/bjm/exploring_the_idea_space_efficiently/):

*   Deliberately avoid exposing yourself to existing lines of thought on how to solve a problem. (The idea here is to defeat anchoring and the availability heuristic.) So **don't review lukeprog's series or read the comments on this thread before generating ideas**.
*   Start by identifying broad categories where ideas might be found. If you're trying to think of calculus word problems, your broad categories might be "jobs, personal life, the natural world, engineering, other".
*   With these initial broad categories, try to include all the categories that might contain a solution and none that will not.
*   Then generate subcategories. Subcategories of "jobs" might include "agriculture, teaching, customer service, manufacturing, research, IT, other". You're also encouraged to generate subsubcategories and so on.
*   Spend more time on those categories that seem promising.
*   You may wish to map your categories and subcategories on a piece of paper.

If you don't like that approach, [here's another](/lw/cja/how_to_brainstorm_effectively/) that's more difficult to summarize. Of course, unstructured idea generation is fine too.

If you're strictly a lurker, you can send your best ideas to lukeprog anonymously using [his feedback box](https://docs.google.com/spreadsheet/viewform?formkey=dG1tUklRelVZdkJscThTV3dKZ1NJb0E6MQ). Or send them to me anonymously using [my feedback box](http://www.admonymous.com/johnmaxwell) so I can post them here and get all your karma.

### Thread Usage

Please [reply here](/r/discussion/lw/d34/brainstorming_additional_ideas_on_how_to_purchase/6tr1) if you wish to comment on the idea of this thread.

You're encouraged to discuss the ideas of others in addition to coming up with your own ideas.

If you split your ideas into individual comments, they can be voted on individually and you will probably increase your karma haul.
# How confident is your atheism?

A friend recently asked how strongly I believe that my deconversion from Christianity was not a mistake. Here's my response, and for those of you who are not Christians, I'm just wondering what numbers you would give:

* * *

"There is a part of me that wants to say the chance is far less than 1 percent. But when I consider what 1% must mean about my ability to follow complex arguments and base my judgement on the right premises, it seems absurd to say that.

Trying to honestly estimate the chance that I'm wrong about the Bible being generally reliable is a fascinating exercise... I know the number is low, but I'm not sure how low.  

Today I would give myself a 1 in 20 chance of being wrong. If I were to consider the arguments of 20 other groups similar to Christian theologians, I would probably misunderstand them at least 1 time in 20. After talking with 20 groups that have a very different worldview, I might think they are all are mistaken, but once in a while, maybe 5% of the time, it would actually be me.

Wow, 5%!?! If I convert that into _"There is a 5% probability that the God of the Bible exists and will send me to hell"_, I feel scared. But I know how to cheer myself up: I just say, _"No way, the chance I'll end up in hell MUST be less than 5%. After all, the God of the Bible is CLEARLY just a big, [mean](http://commonsenseatheism.com/wp-content/uploads/2011/01/Thank-you-Jesus.png "Thank you Jesus!") alpha-monkey and... \[[rehearse all the atheistic arguments here](/lw/ik/one_argument_against_an_army/ "One Argument Against an Army")\]"_.

This back-and-forth from certainty to uncertainty makes me feel like I'm doing something seriously wrong.

So what about you? What chance do you place on some variant of Christianity turning up to be true, and what chance do you think a god of some sort exists?"

Numbers please.
# What's the best way to rest?

So, it's well-known (or, at least, oft-thought) that you can't just work 16 hours a day; if you want to get stuff done, you need to _rest_ from time to time. You have to take breaks.

Today, just now, I realized that I don't really know what the best way to rest is. If I want to rest, should I do something that's fun and interesting, like reading a fantasy novel? Should I do something that's _boring_, like building roads in Minecraft, so that work will seem comparatively interesting when I get back? (And besides, reading a fantasy novel is a bit of a challenge, at least for me, whereas building roads in Minecraft is trivial.) Physical activity probably helps, but how much? Do relaxing activities, like taking a warm shower, help more than just sitting there? How can I tell when it's time to take a break? How can I tell when it's time to get back to work?

(Suggestions for tags would be appreciated.)
# [Cryonics News] Australian cryonics startup: Stasis Systems Australia update

Potentially of interest to my fellow antipodean LessWrongers. [Stasis Systems Australia](http://www.stasissystemsaustralia.com "Stasis Systems Australia") is a company seeking to start a cryonics facility in Australia. Their website is pretty sparse, but they just sent out a mailing list update on how they are going, and it doesn't appear that the information therein is located in any news section of the website, so I thought I'd post it here.

> Hello,
> 
> We at  [Stasis Systems Australia Ltd](http://www.stasissystemsaustralia.com) are happy to report our plans to build a cryonics facility in Australia are progressing well.
> 
> You may remember contacting us or attending our online meeting last year, but here's a quick reminder of what we're doing.
> 
> We are a group of Australasian cryonicists putting together a non-for-profit organisation to build and run the first cryonic storage facility in the southern hemisphere.
> 
> We're proud to have WA-based Marta Sandberg on the board of directors as an advisor, as she has a wealth of knowledge and experience from her ongoing role as a director of the  well-established Cryonics Institute.
> 
> We are now officially incorporated as a not-for-profit company, and one investor away from the magic number of ten that will trigger the next stage of the project - selecting a piece of land and starting construction!
> 
> We have had productive discussions with the NSW Department of Health, and are developing positive relationships with the Cryonics Institute, Alcor, and KrioRus.
> 
> We think what we’re doing is worthwhile and in the long term will be of great benefit to the Australasian community.  If you’d like to get involved either as an investor or a volunteer, that would be fantastic.  We’d love help with articles for the website, search engine optimisation, web graphics, or any other skill you have that we might need.
> 
> We would especially appreciate you passing this update on to anyone you know who might be interested.
> 
> Best regards,
> 
> The Stasis Systems Australia team
> 
> [www.StasisSystemsAustralia.com](http://www.stasissystemsaustralia.com)
> 
> Follow [@StaSysAus](https://twitter.com/#%21/StaSysAus) on Twitter
> 
> Like [StaSysAus](http://www.facebook.com/StaSysAus) on Facebook
# Arrison, Vassar and de Grey speaking at Peter Thiel's startup class [link]

http://blakemasters.tumblr.com/post/25149261055/peter-thiels-cs183-startup-class-19-notes-essay

as someone mentioned the rest of Thiel's notes are also an interesting read

http://blakemasters.tumblr.com/peter-thiels-cs183-startup
# Suggest alternate names for the "Singularity Institute"

Once, a smart potential supporter stumbled upon the [Singularity Institute](http://intelligence.org/)'s (old) website and wanted to know if our mission was something to care about. So he sent our [concise summary](http://intelligence.org/summary/) to an AI researcher and asked if we were serious. The AI researcher saw the word 'Singularity' and, apparently without reading our concise summary, sent back a critique of Ray Kurzweil's "accelerating change" technology curves. (Even though SI researchers tend to be Moore's Law agnostics, and our concise summary says nothing about accelerating change.)

Of course, the 'singularity' we're talking about at SI is [intelligence explosion, not accelerating change](http://facingthesingularity.com/), and intelligence explosion doesn't depend on accelerating change. The term "singularity" _used_ to mean intelligence explosion (or "the arrival of machine superintelligence" or "an event horizon beyond which we can't predict the future because something smarter than humans is running the show"). But with the success of _[The Singularity is Near](http://www.amazon.com/The-Singularity-Is-Near-Transcend/dp/0143037889/)_ in 2005, most people know "the singularity" as "accelerating change."

How often do we miss out on connecting to smart people because they think we're arguing for Kurzweil's curves? One friend in the U.K. told me he never uses the world "singularity" to talk about AI risk because the people he knows thinks the "accelerating change" singularity is "a bit mental." 

LWers are likely to have attachments to the word 'singularity,' and the term _does_ often mean intelligence explosion in [the](http://singularityhypothesis.blogspot.com/) [technical](http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf) [literature](http://consc.net/papers/singularityjcs.pdf), but neither of these is a strong reason to keep the word 'singularity' in the name of our AI Risk Reduction organization. If the 'singularity' term is keeping us away from many of the people we care most about reaching, maybe we should change it.

Here are some possible alternatives, without trying too hard:

*   The Center for AI Safety
*   The I.J. Good Institute
*   Beneficial Architectures Research
*   A.I. Impacts Research

**We almost certainly won't change our name within the next year**, but it doesn't hurt to start gathering names now and do some market testing. You were all [very helpful](/lw/9lx/help_name_suggestions_needed_for_rationalityinst/) in naming "Rationality Group". (BTW, the winning name, "Center for Applied Rationality," came from LWer [beoShaffer](/lw/9lx/help_name_suggestions_needed_for_rationalityinst/5s0k).)

And, before I am vilified by people who have as much positive affect toward the name "Singularity Institute" as I do, let me note that this was _not originally my idea_, but I do think it's an idea worth taking seriously enough to bother with some market testing.
# Ekman Training - Reviews and/or Testing

I'm considering taking Ekman's [microexpressions training](https://face.paulekman.com/face/products.aspx) because it's cheap in both time and money. Has anyone here taken it? Did it work for you? How do you know?

The course does seem to come with tests included (both before and after), but if anyone has any ideas for some **cheap** tests I can do before and after to see if it really works, I'd be happy to do those as well, and report the results. Cheap tests should cost me less than three hours total and less than $100 total.

Alternately, if enough people here have done it we could pool our "before" and "after" scores to independently verify whether there's an effect.