Exploring Moral Reasoning in a Cognitive Architecture

Exploring Moral Reasoning in a Cognitive Architecture Wayne Iba Pat Langley Computer Science Department Westmont College, 955 La Paz Road Santa Barbara, CA 93108 Institute for the Study of Learning and Expertise 2164 Staunton Court, Palo Alto, CA 94306 Abstract Background Moral reasoning plays a significant but poorly under- stood role in human action and interaction. Although studied by philosophers for millennia, considerable con- fusion surrounds the topic. Computational cognitive ar- chitectures hold promise for shedding insight on how agents act and reason morally. We present a view of moral cognition and examine one implementation of that view in Icarus, a theory of the human cognitive archi- tecture. This approach to moral behavior and reasoning leads us to suggest that morality is a special case of ev- eryday cognition. We discuss the implications of this view and outline our continuing research on these and related questions. Keywords: Machine ethics; Moral reasoning; Philoso- phy; Cognitive architectures. In this paper, we describe our attempt to model moral cognition within an integrated cognitive architecture. As such, we draw upon work in three distinct disciplines – philosophy, psychology and computer science – each of which has its own body of literature. In this section we point to a very small sample of work that has influenced our thinking. Although a distinction may be made between ethics and morals, for our present purposes we will use the terms interchangeably. The point of an ethical theory is to provide guidelines for how to live “the good life”. As such, an ethical theory should identify the nature of the good as well as provide a system for acting so as to achieve the good. Here we are not advancing a particular theory, but rather exploring how a specific cognitive architecture could support moral reasoning of different types. However, it will help readers to have at least a basic familiarity of the main ethical theories for discussions in later sections. The three dominant ethical theories and their primary initiators include the virtue ethics of Aristotle, deonto- logical systems descending from Kant, and consequen- tialist theories following from Hume, Bentham, and Mill. In virtue ethics, one lives the good life by behaving in a balanced manner, thereby exhibiting the virtues – nei- ther too much nor too little of a given quality. Deon- tological systems place value on fulfilling one’s duties or obligations; commonly, this is viewed as following cer- tain rules that prescribe how one ought to behave. Con- sequentialist theories assign different values to different states and prescribe courses of action that result in states with the best values; utilitarianism is one particular form of consequentialism where actions are selected so as to maximize the total good across the entire population. One approach to understanding moral reasoning is to observe and explain what people do when they are faced with morally charged decisions (Baron & Ritov, 2009; Waldmann & Dieterich, 2007). In contrast to the philosophical theories that prescriptively indicate what “ought” to be done, this approach attempts to charac- terize what actually takes place and why. Such studies provide hints and constraints on determining the nature of moral decision making in humans (Spranca, Minsk, & Baron, 1991). For example, many of these studies sug- gest that people operate using a hybrid of deontic and Introduction Moral reasoning has been a focus of study for millennia. The human condition stems in large part from the col- lective judgments and decisions that can be said to have moral valence. This suggests that it would be desir- able to study this topic carefully. Nevertheless, our un- derstanding of morally charged cognition is still incom- plete and uncertain. Since the introduction of comput- ers, many people have been fascinated with the prospect of writing programs that exhibit human intelligence. As with other aspects of intelligence, computational mod- els can provide many insights into the nature of moral cognition. Although generally treated as a topic distinct from generic reasoning, we suggest here that moral cogni- tion is better viewed as a special case of practical cog- nition. Depending on the ethical approach one takes, moral dilemmas might be nothing other than the conse- quence of bounded rationality. Perhaps our subjective experience when deliberating over a moral dilemma has more to do with an emotion than with the properties of the problem that we are trying solve. These two factors – bounded rationality and emotional states – may be the actual source of moral dilemmas. We start by distinguishing between three types of moral cognition: moral behavior, moral interpretation, and moral decision making. Next we review Icarus, the cognitive architecture we have used to implement agents that exhibit forms of moral cognition. We discuss our experiences with these agents and their broader implica- tions for accounts of this class of phenomena. In closing, we suggest directions for future research that should fur- ther clarify our understanding of moral cognition.