# Beginning resources for CEV research

I've been working on [metaethics/CEV research](http://wiki.lesswrong.com/wiki/No-Nonsense_Metaethics) for a couple months now (publishing mostly [prerequisite](http://wiki.lesswrong.com/wiki/Intuitions_and_Philosophy) [material](/lw/4z7/the_neuroscience_of_desire/)) and figured I'd share some of the sources I've been using.

**CEV sources**.

*   Yudkowsky, [Metaethics sequence](http://wiki.lesswrong.com/wiki/Metaethics_sequence)
*   Yudkowsky, '[Coherent Extrapolated Volition](http://intelligence.org/upload/CEV.html)'
*   Tarleton, '[Coherent extrapolated volition: A meta-level approach to machine ethics](http://intelligence.org/upload/coherent-extrapolated-volition.pdf)'

**Motivation**. CEV extrapolates human motivations/desires/values/volition. As such, it will help to understand how human motivation works.

*   _Neuroeconomics_ studies motivation as a driver of action under uncertainty. Start with _[Neuroeconomics: Decision Making and the Brain](http://www.amazon.com/Neuroeconomics-Decision-Paul-W-Glimcher/dp/0123741769/)_ (2008) and _[Foundations of Neuroeconomic Analysis](http://www.amazon.com/dp/0199744254/)_ (2010), and see my bibliography [here](/lw/4z7/the_neuroscience_of_desire/).
*   _Affective neuroscience_ studies motivation as an emotion. Start with _[Pleasures of the Brain](http://www.amazon.com/Pleasures-Affective-Science-Morten-Kringelbach/dp/0195331028/)_ (2009) and my bibliography [here](/lw/4yq/the_neuroscience_of_pleasure/).
*   _Motivation science_ integrates psychological approaches to studying motivation. Start with _[The Psychology of Goals](http://www.amazon.com/Psychology-Goals-Gordon-Moskowitz-PhD/dp/1606230298/)_ (2009), _[Oxford Handbook of Human Action](http://www.amazon.com/Oxford-Handbook-Action-Cognition-Neuroscience/dp/0195309987/)_ (2008), and _[Handbook of Motivation Science](http://www.amazon.com/Handbook-Motivation-Science-James-Shah/dp/1593855680/)_ (2007).

  
**Extrapolation**. Is it plausible to think that some kind of extrapolation of human motivations will converge on a single motivational set? How would extrapolation work, exactly?

*   _Reflective equilibrium_. Yudkowsky's proposed extrapolation works analogously to what philosophers call '[reflective equilibrium](http://plato.stanford.edu/entries/reflective-equilibrium/).' The most thorough work here is the [1996 book](http://www.amazon.com/Justice-Justification-Reflective-Equilibrium-Philosophy/dp/052146711X/) by Daniels, and there have been [lots of papers](http://philpapers.org/browse/reflective-equilibrium), but this genre is only barely relevant for CEV. Basically, an entirely new literature on volition-extrapolation algorithms needs to be created.
*   _Full-information accounts of value_ and _ideal observer_ theories. This is what philosophers call theories of value that talk about 'what we would want if we were fully informed, etc.' or 'what a perfectly informed agent would want' like CEV does. There's [some](http://scholar.google.com/scholar?as_q=full+information+accounts&num=10&btnG=Search+Scholar&as_epq=&as_oq=value+good&as_eq=&as_occt=any&as_sauthors=&as_publication=&as_ylo=&as_yhi=&as_sdt=1&as_subj=soc&as_sdtf=&as_sdts=5&hl=en) [literature](http://en.wikipedia.org/wiki/Ideal_observer_theory) on this, but it's only marginally relevant to CEV. Again, an entirely new literature needs to be written to solve this problem.

  
**Metaethics**. Should we use CEV, or something else? What does 'should' mean?

*   Yudkowsky, [Metaethics sequence](http://wiki.lesswrong.com/wiki/Metaethics_sequence)
*   _[An Introduction to Contemporary Metaethics](http://www.amazon.com/Introduction-Contemporary-Metaethics-Alex-Miller/dp/074562345X/)_ is a good introduction to mainstream metaethics. Unfortunately, nearly all of mainstream metaethics is horribly misguided, but the book will at least give you a good sense of the questions involved and what some of the wrong answers are. The chapter on moral reductionism is the most profitable.
*   Also see '[Which Consequentialism? Machine ethics and moral divergence](http://bentham.k2.t.u-tokyo.ac.jp/ap-cap09/openconf/data/papers/33.pdf).'

  
**Building the utility function**. How can a seed AI be built? How can it read what to value?

*   Dewey, '[Learning What to Value](http://www.danieldewey.net/dewey-learning-what-to-value.pdf)'
*   Yudkowsky, '[Coherent Extrapolated Volition](http://intelligence.org/upload/CEV.html)'
*   Yudkowsky, '[Artificial Intelligence as a Positive and Negative Factor in Global Risk](http://intelligence.org/upload/artificial-intelligence-risk.pdf)'

  
**Preserving the utility function**. How can the motivations we put into a superintelligence be preserved over time and self-modifcation?

*   Yudkowsky, '[Coherent Extrapolated Volition](http://intelligence.org/upload/CEV.html)'
*   De Blanc, '[Ontological Crises in Artificial Agents' Value Systems](http://arxiv.org/pdf/1105.3821v1)'
*   Omohundro, '[Basic AI Drives](http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf)' and '[The Nature of Self-Improving Artificial Intelligence](http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf)' (instrumental drives to watch out for, and more)

  
**Reflective decision theory**. Current decision theories tell us little about software agents that make decisions to modify their own decision-making mechanisms.

*   See the Less Wrong wiki page on [decision theory](http://wiki.lesswrong.com/wiki/Decision_theory).
*   Wei Dai's [Updateless Decision Theory](http://wiki.lesswrong.com/wiki/Updateless_decision_theory)
*   Yudkowsky's [Timeless Decision Theory](http://wiki.lesswrong.com/wiki/Timeless_decision_theory)

  
Additional suggestions welcome. I'll try to keep this page up-to-date.