MIRI

MACH IN E INT ELLIGENCER ESEARCH INS TITU TE

Exploratory Engineering in AI
Luke Muehlhauser
Machine Intelligence Research InstituteBill Hibbard
University of Wisconsin
Madison Space Science and Engineering Center

Muehlhauser, Luke and Bill Hibbard. 2013. “Exploratory Engineering in AI”
Communications of the ACM, Vol. 57 No. 9, Pages 32–34. doi:10.1145/2644257
This version contains minor changes.

We regularly see examples of new artificial intelligence (AI) capabilities. Google’s selfdriving car has safely traversed thousands of miles. Watson beat the Jeopardy! champions, and Deep Blue beat the chess champion. Boston Dynamics’ Big Dog can walkover uneven terrain and right itself when it falls over. From many angles, software canrecognize faces as well as people can.
As their capabilities improve, AI systems will become increasingly independent ofhumans. We will be no more able to monitor their decisions than we are now able tocheck all the math done by today’s computers. No doubt such automation will produce
tremendous economic value, but will we be able to trust these advanced autonomous
systems with so much capability?
For example, consider the autonomous trading programs which lost Knight Capital$440 million (pre-tax) on August 1st, 2012, requiring the firm to quickly raise $400 million to avoid bankruptcy (Valetkevitch and Mikolajczak 2012). This event undermines
a common view that AI systems cannot cause much harm because they will only ever
be tools of human masters. Autonomous trading programs make millions of trading
decisions per day, and they were given suﬃcient capability to nearly bankrupt one of thelargest traders in U.S. equities.
Today, AI safety engineering mostly consists in a combination of formal methods
and testing. Though powerful, these methods lack foresight: they can be applied only
to particular extant systems. We describe a third, complementary approach which aims
to predict the (potentially hazardous) properties and behaviors of broad classes of future
AI agents, based on their mathematical structure (e.g. reinforcement learning). Suchprojects hope to discover methods “for determining whether the behavior of learning
agents [will remain] within the bounds of pre-specified constraints . . . after learning”(Spears 2006). We call this approach “exploratory engineering in AI.”

1. Exploratory Engineering in Physics, Astronautics, Computing,and AI
In 1959, Richard Feynman pointed out that the laws of physics (as we understand them)straightforwardly imply that we should be able to “write the entire 24 volumes of theEncyclopaedia Brittanica on the head of a pin” (Feynman 1959). Feynman’s aim was todescribe technological possibilities as constrained not by the laboratory tools of his day
but by known physical law, a genre of research Eric Drexler later dubbed “exploratoryengineering” (Drexler 1991). Exploratory engineering studies the ultimate limits of yetto-be-engineered devices, just as theoretical physics studies the ultimate limits of naturalsystems. Thus, exploratory engineering “can expose otherwise unexpected rewards frompursuing particular research directions [and] thus improve the allocation of scientificresources” (Drexler 1992).
This kind of exploratory engineering in physics led to large investments in nanoscaletechnologies and the creation of the U.S. National Nanotechnology Initiative. Today,
nanoscale technologies have a wide range of practical applications, and in 2007 Israeli
scientists printed the entire Hebrew Bible onto an area smaller than the head of a pin(Associated Press 2007).
Nanoscience is hardly the only large-scale example of exploratory engineering.
Decades earlier, the scientists of pre-Sputnik astronautics studied the implications ofphysical law for spaceflight, and their analyses enabled the later construction and launchof the first spacecraft. In the 1930s, Alan Turing described the capabilities and limitations of mechanical computers several years before John von Neumann, Konrad Zuse,
and others figured out how to build them. And since the 1980s, quantum computing
researchers have been discovering algorithms and error-correction techniques for quantum computers that we cannot yet build—but whose construction is compatible withknown physical law.
Pushing the concept of exploratory engineering a bit beyond Drexler’s original definition, we apply it to some recent AI research that formally analyzes the implications ofsome theoretical AI models. These models might not lead to useful designs as was thecase in astronautics and nanoscience, but like the theoretical models that Butler Lampson used to identify the “confinement problem” in 1973 (Lampson 1973), these theoretical AI models do bring to light important considerations for AI safety, and thus they
“expose otherwise unexpected rewards from pursuing particular research directions” inthe field of AI safety engineering. In this article, we focus on theoretical AI models inspired by Marcus Hutter’s AIXI (Hutter 2012), an optimal agent model for maximizing
an environmental reward signal.

2. AIXI-like Agents and Exploratory Engineering
How does AIXI work? Just as an idealized chess computer with vast amounts of computing power could brute-force its way to perfect chess play by thinking through theconsequences of all possible move combinations, AIXI brute-forces the problem of general intelligence by thinking through the consequences of all possible actions, given allpossible ways the universe might be. AIXI uses Solomonoﬀ ’s universal prior to assign a
relative prior probability to every possible (computable) universe, marking simpler hypotheses as more likely. Bayes’ Theorem is used to update the likelihood of hypotheses
based on observations. To make decisions, AIXI chooses actions that maximize its ex-pected reward. More general variants of AIXI maximize a utility function defined on
their observations and actions.
Based on an assumption of a stochastic environment containing an infinite amountof information, the original AIXI model is uncomputable and therefore not a subjectof exploratory engineering. Instead, finitely computable variants of AIXI, based on theassumption of a stochastic environment containing a finite amount of information, canbe used for exploratory engineering in AI. The results described below don’t depend on
the assumption of infinite computation.
A Monte-Carlo approximation of AIXI can play Pac-Man and other simple games(Veness et al. 2011), but some experts think AIXI approximation isn’t a fruitful path
toward human-level AI. Even if that’s true, AIXI is the first model of cross-domain
intelligent behavior to be so completely and formally specified that we can use it tomake formal arguments about the properties which would obtain in certain classes ofhypothetical agents if we could build them today. Moreover, the formality of AIXI-likeagents allows researchers to uncover potential safety problems with AI agents of increasingly general capability—problems which could be addressed by additional research, as
happened in the field of computer security after Lampson’s article on the confinement
problem.
AIXI-like agents model a critical property of future AI systems: that they will needto explore and learn models of the world. This distinguishes AIXI-like agents fromcurrent systems that use predefined world models, or learn parameters of predefined
world models. Existing verification techniques for autonomous agents (Fisher, Dennis,and Webster 2013) apply only to particular systems, and to avoiding unwanted optima
in specific utility functions. In contrast, the problems described below apply to broadclasses of agents, such as those that seek to maximize rewards from the environment.
For example, in 2011 Mark Ring and Laurent Orseau analyzed some classes of AIXIlike agents to show that several kinds of advanced agents will maximize their rewards
by taking direct control of their input stimuli (Ring and Orseau 2011). To understandwhat this means, recall the experiments of the 1950s in which rats could push a leverto activate a wire connected to the reward circuitry in their brains. The rats pressed thelever again and again, even to the exclusion of eating. Once the rats were given directcontrol of the input stimuli to their reward circuitry, they stopped bothering with moreindirect ways of stimulating their reward circuitry, such as eating. Some humans also
engage in this kind of “wireheading” behavior when they discover that they can directly
modify the input stimuli to their brain’s reward circuitry by consuming addictive narcotics. What Ring and Orseau showed was that some classes of artificial agents will
wirehead—that is, they will behave like drug addicts.

Fortunately, there may be some ways to avoid the problem. In their 2011 paper, Ringand Orseau showed that some types of agents will resist wireheading. And in 2012,
Bill Hibbard (2012) showed that the wireheading problem can also be avoided if three
conditions are met: (1) the agent has some foreknowledge of a stochastic environment,(2) the agent uses a utility function instead of a reward function, and (3) we define
the agent’s utility function in terms of its internal mental model of the environment.
Hibbard’s solution was inspired by thinking about how humans solve the wireheading
problem: we can stimulate the reward circuitry in our brains with drugs, yet most of us
avoid this temptation because our models of the world tell us that drug addiction willchange our motives in ways that are bad according to our current preferences.
Relatedly, Daniel Dewey (2011) showed that in general, AIXI-like agents will locate
and modify the parts of their environment that generate their rewards. For example,
an agent dependent on rewards from human users will seek to replace those humans
with a mechanism that gives rewards more reliably. As a potential solution to this problem, Dewey proposed a new class of agents called value learners, which can be designed
to learn and satisfy any initially unknown preferences, so long as the agent’s designers
provide it with an idea of what constitutes evidence about those preferences.
Practical AI systems are embedded in physical environments, and some experimental
systems employ their environments for storing information. Now AIXI-inspired workis creating theoretical models for dissolving the agent-environment boundary used asa simplifying assumption in reinforcement learning and other models, including theoriginal AIXI formulation (Orseau and Ring 2012b). When agents’ computations mustbe performed by pieces of the environment, they may be spied on or hacked by other,
competing agents. One consequence shown in another paper by Orseau and Ring isthat, if the environment can modify the agent’s memory, then in some situations eventhe simplest stochastic agent can outperform the most intelligent possible deterministicagent (Orseau and Ring 2012a).

3. Conclusion
Autonomous intelligent machines have the potential for large impacts on our civilization(Vardi 2012). Exploratory engineering gives us the capacity to have some foresight into
what these impacts might be, by analyzing the properties of agent designs based on
their mathematical form. Exploratory engineering also enables us to identify lines ofresearch—such as the study of Dewey’s value-learning agents—that may be important
for anticipating and avoiding unwanted AI behaviors. This kind of foresight will be
increasingly valuable as machine intelligence comes to play an ever-larger role in ourworld.

References
Associated Press. 2007. “Haifa Technion Scientists Create World’s Smallest Bible.” December 24. http://www.haaretz.com/news/haifa-technion-scientists-create-world-s-smallestbible-1.235825.
Bach, Joscha, Ben Goertzel, and Matthew Iklé, eds. 2012. Artificial General Intelligence: 5th International
Conference, AGI 2012, Oxford, UK, December 8–11, 2012. Proceedings. Lecture Notes in Artificial
Intelligence 7716. New York: Springer. doi:10.1007/978-3-642-35506-6.
Dewey, Daniel. 2011. “Learning What to Value.” In Schmidhuber, Thórisson, and Looks 2011, 309–314.
Drexler, K. Eric. 1991. “Exploring Future Technologies.” In Doing Science: The Reality Club, edited byJohn Brockman, 129–150. New York: Prentice Hall.
. 1992. Nanosystems: Molecular Machinery, Manufacturing, and Computation. New York: Wiley.
Feynman, Richard P. 1959. “There’s Plenty of Room at the Bottom.” Lecture given at the Annual Meetingof the American Physical Society, Pasadena, CA, December 29. http://calteches.library.
caltech.edu/47/2/1960Bottom.pdf.
Fisher, Michael, Louise Dennis, and Matt Webster. 2013. “Verifying Autonomous Systems.” Communications of the ACM 56 (9): 84–93. doi:10.1145/2494558.
Hibbard, Bill. 2012. “Model-Based Utility Functions.” Journal of Artificial General Intelligence 3 (1): 1–24.
doi:10.2478/v10229-011-0013-5.
Hutter, Marcus. 2012. “One Decade of Universal Artificial Intelligence.” In Theoretical Foundations ofArtificial General Intelligence, edited by Pei Wang and Ben Goertzel. Atlantis Thinking Machines 4.
Paris: Atlantis.
Lampson, Butler W. 1973. “A Note on the Confinement Problem.” Communications of the ACM 16 (10):613–615. doi:10.1145/362375.362389.
Orseau, Laurent, and Mark Ring. 2012a. “Memory issues of Intelligent Agents.” In Bach, Goertzel, andIklé 2012, 219–231.
. 2012b. “Space-Time Embedded Intelligence.” In Bach, Goertzel, and Iklé 2012, 209–218.
Ring, Mark, and Laurent Orseau. 2011. “Delusion, Survival, and Intelligent Agents.” In Schmidhuber,Thórisson, and Looks 2011, 11–20.
Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. 2011. Artificial General Intelligence:
4th International Conference, AGI 2011, Mountain View, CA, USA, August 3–6, 2011. Proceedings.
Lecture Notes in Computer Science 6830. Berlin: Springer. doi:10.1007/978-3-642-22887-2.
Spears, Diana F. 2006. “Assuring the Behavior of Adaptive Agents.” In Agent Technology from a Formal Perspective, edited by Christopher Rouﬀ, Michael Hinchey, James Rash, Walter Truszkowski,and Diana F. Gordon-Spears, 227–257. NASA Monographs in Systems and Software Engineering.
London: Springer. doi:10.1007/1-84628-271-3_8.
Valetkevitch, Caroline, and Chuck Mikolajczak. 2012. “Error by Knight Capital Rips Through Stock
Market.” Reuters, September 25. http://www.reuters.com/article/2012/08/01/us-usanyse-tradinghalts-idUSBRE8701BN20120801.
Vardi, Moshe Y. 2012. “The Consequences of Machine Intelligence.” The Atlantic, October 25. http:/ / www . theatlantic . com / technology / archive / 2012 / 10 / the - consequences - of machine-intelligence/264066/.

Veness, Joel, Kee Siong Ng, Marcus Hutter, William Uther, and David Silver. 2011. “A Monte-CarloAIXI Approximation.” Journal of Artificial Intelligence Research 40:95–142. doi:10 . 1613 / jair .
3125.