Requirements for an Artificial Agent with Norm Competence

Human behavior is frequently guided by social and moral norms, and no human community can exist without norms. Robots that enter human societies must therefore behave in norm-conforming ways as well. However, currently there is no solid cognitive or computational model available of how human norms are represented, activated, and learned. We provide a conceptual and psychological analysis of key properties of human norms and identify the demands these properties put on any artificial agent that incorporates norms— demands on the format of norm representations, their structured organization, and their learning algorithms. No human community can exist without norms (Hechter and Opp 2001; Ullmann-Margalit 1977), and many past human communities have gone extinct with suboptimal systems of norms (Wilson 2002). It stands to reason that communities that include both humans and machines as partners will also not succeed without norms. If this is true then we need to understand and formalize what norms are in the human mind—how people represent, learn, activate, update, and deploy norms to guide their behavior—so that we can effectively design artificial agents with appropriate capacities to represent and obey norms. If an artificial agent is to acquire human norms, its formalisms and algorithms must be informed by the properties of human norms—how humans represent norms, learn them, and use them to guide behavior. We introduce here core properties of human norms and define the demands these properties put on any artificial agent that incorporates norms. These demands range from the format of norm representations to their structured organization, from learning algorithms to communication skills. In sociology and experimental economics, the importance of norms has long been recognized (Parsons 1951; Schelling 1960). These literatures try to explain human cooperation despite the individual’s rational self-interest, and norms are an external force that constrains human action. But it is not known how such external forces can operate cognitively and computationally. A person complying with norms must have something in their mind that allows their action to conform to the norm, and that something may be called a norm representation. A sizable set of studies has examined the automatic activation of such norm representations by situation cues—for example, garbage on the floor triggers the “don’t litter” norm (Cialdini, Reno, and Kallgren 1990), or the sight of a library triggers the “be quiet” norm (Aarts and Dijksterhuis 2003). But no cognitive model has been offered that specifies at least some of the key properties of norm representations. This is what we attempt to do here. Properties of Human Norms Working Definition of Norm We adopt the following working definition of a norm ([citation concealed]; cf. Bicchieri 2006; Brennan et al. 2013): A norm is an instruction to (not) perform a type of action in a given context, provided that a sufficient number of individuals in a community (i) indeed follow this instruction and (ii) demand of each other to follow the instruction. This definition captures both the “external” aspect of norms (that they are obeyed and enforced by communities) and their “internal” aspect (that they guide actions). Accordingly, “instruction” can sometimes refer to one person or group literally instructing another person, sometimes to a self-instruction that motivates a person to perform a certain action. The term action covers a broad class—including physical (observable) or mental acts, omissions, as well as acts of bringing about a certain outcome. The separation into conditions (i) and (ii) follows a long tradition of considering both “descriptive” and “injunctive” elements of norms (Bicchieri 2006; Gibbs 1965; Cialdini, Reno, and Kallgren 1990). We can call condition (i) the prevalence component of a norm—that members of a community do in fact follow a norm (with a certain degree of consistency); and we can call condition (ii) the normative demand component of a norm— the degree to which community members demand of one another that each follow the norm. Because of this normative demand, violations of norms often lead to sanctions (e.g., criticism, isolation, reform, punishment). However, the existence of sanctions need not be part of the definition of a norm, as some have argued (e.g., Bendor and Swistak 2001). It is entirely conceivable that for some norm in some communities, no sanctions have been necessary to uphold a norm. That would obviously not make it any less of a norm. For something to be a norm requires that people in a relevant community meet conditions (i) and (ii). For someone to have a norm representation requires that the agent knows conditions (i) and (ii); and for someone to show norm compliance, the agent both knows conditions (i) and (ii) and tries to follow the instruction because of conditions (i) and (ii). Related concepts. Norms differ from other action guides, such as preferences, goals, and collective habits. The normative force condition marks this difference. A lot of people put milk in their coffee, but they do not demand it of each other, so this action is not a norm but a wide-spread preference. By contrast, getting in line to order coffee is a norm, because that is what people would expect of each other. Norms also differ from values, and the notion of being instructions to act in a particular context marks this difference. Values (such as fairness, freedom, dignity) typically govern a larger class of possible actions/outcomes across a wider range of contexts. Implied and Suggested Properties We discuss now six properties of norms, either implied by or further elaborated from the working definition, and develop demands that these properties put on artificial agents’ norm representations. 1. Multiple norm types A first property is that norms can be of multiple deontic types: at least prescriptions, prohibitions, and permissions. Thus, the representation of a norm-guided action must signal which of the types governs the particular action. 2. Context sensitivity A second critical property of norms is that they are contextspecific and must somehow be activated by characteristic features of a given context. In initial research we asked people to state the prescription norms that apply for a variety of everyday scenes (e.g., board room, jogging path), and 95% of these norms were specific to the context [cit. concealed]. 3. Community prevalence A third property is that genuine norms have sufficient prevalence—that is, most community members comply with a given norm and are aware of that collective compliance. In initial research we found that the top 8 prescription or prohibition norms that people stated for various everyday scenes showed prevalence rates between 94% (most prevalent) and 21% (8th-most prevalent). Beyond the actual agreement rates, people also have a belief about the norm’s prevalence in the community, which we can call the prevalence parameter. Though this belief is not necessarily accurate (Prentice & Miller, 1996), it will often be based on reasonably representative behavioral data. 4. Graded normative demand A rarely noted property is that (at least) prescriptions and prohibitions come in degrees of community demand [cit. concealed]. At least in English, terms of prescription can capture low demand (“it is suggested to A”) to high demand (“it is required to A”), with further gradations in between. Likewise, terms of prohibition can capture low demand (“it is frowned upon to A”) to high demand (“it is forbidden to A”). In preliminary research we found that people show high consensus in ordering these terms along a dimension of normative demand, and Figure 1 shows such orderings for all three norm types [cit. concealed]. Thus, human norm representations include a graded normative demand parameter. A norm’s degree of normative demand is likely to be closely related to its prevalence, but the two are not reducible to each other. In general, the more strongly people demand of others to conform to norm Ni, the more people will obey it. But when community norms change, strong demand sometimes lingers even though prevalence is declining (Mack 2018); and for some norms of only modest normative demand, prevalence may be high (e.g., if the benefits of norm conformity are considerable). Even though the exact relationship between prevalence and normative demand is unknown, as a first approximation we can assume that demand is a linear function of prevalence and other factors such as severity of consequences. 5. Resolving norm conflict by normative demand Sometimes norms stand in conflict with one another such that, in the given context, every action violates at least one norm (e.g., in moral dilemmas). Because normative demand comes in degrees, violating some norms will be more costly than violating others, so norm conflict resolution will have to take graded normative demand into account. 6. Structural representation of norms The norms relevant to a particular context must be organized in some way, and one plausible organization is a network structure that has some core (strong or consensual) norms at the center and other (weaker or less consensual) norms in Figure 1. Linguistic terms referring to degrees of normative demand for prescriptions (left), prohibitions (right), and permissions (middle). Figure 2. Network representation of prescriptions people generated when exposed to the picture of a library. Larger circles indicate norms that were mentioned by more people. the periphery. In our preliminary research we found that norms generated by a sample of participants for specific contexts clustered together in such networks (see Figure 2 as an illustration; [cit. concealed]). We also found that, when exposed to a particular context, norms mentioned earlier were more prevalent—thus, they were likely at the core of the network. Formal Representation Most formal appro