# Connecting Your Beliefs (a call for help)

[![](http://commonsenseatheism.com/wp-content/uploads/2011/11/bayesian-belief-network-small.png)](http://commonsenseatheism.com/wp-content/uploads/2011/11/bayesian-belief-network.jpg)A couple weeks after meeting me, [Will Newsome](/user/Will_Newsome/) gave me one of the best compliments I’ve ever received. He said: “Luke seems to have two copies of the [Take Ideas Seriously](/lw/2l6/taking_ideas_seriously/) gene.”

What did Will mean? To take an idea seriously is “to update a belief and then accurately and completely propagate that belief update through the entire web of beliefs in which it is embedded,” as in a [Bayesian belief network](http://en.wikipedia.org/wiki/Belief_propagation) (see right).

Belief propagation is what happened, for example, when I first encountered that thundering paragraph from [I.J. Good (1965)](http://commonsenseatheism.com/wp-content/uploads/2011/02/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf):

> Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind... Thus the first ultraintelligent machine is the last invention that man need ever make.

Good’s paragraph ran me over like a train. Not because it was absurd, but because it was clearly true. Intelligence explosion was a direct consequence of things I already believed, I just hadn’t noticed! Humans do not automatically propagate their beliefs, so I hadn’t noticed that my worldview already implied intelligence explosion.

I spent a week looking for counterarguments, to check whether I was missing something, and then accepted intelligence explosion to be likely (so long as scientific progress continued). And though I hadn’t read Eliezer on the [complexity of value](/lw/y3/value_is_fragile/), I _had_ read [David Hume](http://en.wikipedia.org/wiki/David_Hume#Ethics) and [Joshua Greene](http://www.wjh.harvard.edu/~jgreene/GreeneWJH/Greene-Dissertation.pdf). So I already understood that an arbitrary artificial intelligence would almost certainly not share our values.

Accepting my belief update about intelligence explosion, I propagated its implications throughout my web of beliefs. I realized that:

*   Things can go _very_ wrong, for we live in a world [beyond the reach of God](/lw/uk/beyond_the_reach_of_god/).
*   Scientific progress can destroy the world.
*   Strong [technological determinism](http://en.wikipedia.org/wiki/Technological_determinism) is true; purely social factors will be swamped by technology.
*   Writing about philosophy of religion was not important enough to consume any more of my time.
*   My highest-utility actions are either those that work toward reducing AI risk, or those that work toward making lots of money so I can _donate_ to AI risk reduction.
*   Moral theory is not idle speculation but an [urgent engineering problem](http://commonsenseatheism.com/?p=14013).
*   Technological utopia is possible, but unlikely.
*   The value of information concerning intelligence explosion scenarios is extremely high.
*   Rationality is even _more_ important than I already believed it was.
*   and more.

I had encountered the I.J. Good paragraph on Less Wrong, so I put my other projects on hold and spent the next month reading almost [everything](http://commonsenseatheism.com/?p=13052) Eliezer had written. I also found articles by [Nick Bostrom](http://nickbostrom.com/) and [Steve Omohundro](http://steveomohundro.com/). I began writing articles for Less Wrong and learning from the community. I applied to [Singularity Institute](http://intelligence.org/)’s Visiting Fellows program and was accepted. I quit my job in L.A., moved to Berkeley, worked my ass off, got hired, and started collecting research related to [rationality](/lw/7e5/the_cognitive_science_of_rationality/) and [intelligence explosion](/lw/71x/a_crash_course_in_the_neuroscience_of_human/).

My story surprises people because it _is_ unusual. Human brains don’t usually propagate new beliefs so thoroughly.

But this isn’t just another post on taking ideas seriously. Will [already offered](/lw/2l6/taking_ideas_seriously/) some ideas on _how_ to propagate beliefs. He also listed some ideas that most people probably aren’t taking seriously enough. My purpose here is to examine one prerequisite of successful belief propagation: _actually making sure your beliefs are connected to each other in the first place_.

If your beliefs aren’t connected to each other, there may be no paths along which you _can_ propagate a new belief update.

I’m not talking about the problem of [free-floating beliefs](http://wiki.lesswrong.com/wiki/Free-floating_belief) that don’t [control your anticipations](/lw/i3/making_beliefs_pay_rent_in_anticipated_experiences/). No, I’m talking about “[proper](http://wiki.lesswrong.com/wiki/Improper_belief)” beliefs that [require observation](http://wiki.lesswrong.com/wiki/Beliefs_require_observations), can be [updated](http://wiki.lesswrong.com/wiki/Belief_update) by evidence, and [pay rent](http://wiki.lesswrong.com/wiki/Beliefs_pay_rent) in anticipated experiences. The trouble is that even _proper_ beliefs can be inadequately connected to other proper beliefs inside the human mind.

I wrote this post because I'm not sure what the "making sure your beliefs are actually connected in the first place" skill _looks like_ when broken down to the [5-second level](/lw/5kz/the_5second_level/).

I was chatting about this with [atucker](/user/atucker/), who told me he noticed that successful businessmen may have this trait more often than others. But what are they doing, at the 5-second level? What are people like Eliezer and [Carl](/lw/7ob/timeline_of_carl_shulman_publications/) doing? How does one engage in the purposeful decompartmentalization of one's own mind?