Abstract: Artificial intelligence (AI) experts are currently divided into "presentist" and "futurist" factions that call for attention to near-term and long-term AI, respectively. This paper argues that the presentist-futurist dispute is not the best focus of attention. Instead, the paper proposes a reconciliation between the two factions based on a mutual interest in AI. The paper further proposes a realignment to two new factions: an "intellectualist" faction that seeks to develop AI for intellectual reasons (as found in the traditional norms of computer science) and a "societalist faction" that seeks to develop AI for the benefit of society. The paper argues in favor of societalism and offers three means of concurrently addressing societal impacts from near-term and long-term AI: (1) advancing societalist social norms, thereby increasing the portion of AI researchers who seek to benefit society; (2) technical research on how to make any AI more beneficial to society; and (3) policy to improve the societal benefits of all AI. In practice, it will often be advantageous to emphasize near-term AI due to the greater interest in near-term AI among AI and policy communities alike. However, presentist and futurist societalists alike can benefit from each others' advocacy for attention to the societal impacts of AI. A reconciliation between the presentist and futurist factions can improve both near-term and long-term societal impacts of AI.

Reconciliation Between Factions Focused on Near-Term and Long-Term Artificial Intelligence

Introduction Artificial intelligence (AI) experts are-to generalize-of two minds about long-term AI, especially regarding the potential for radically transformative types of future AI such as artificial superintelligence (ASI) and brain emulation. One faction views long-term AI as a profound issue, warranting extensive attention and immediate action. The other faction views long-term AI as an unhelpful distraction from the more pressing near-term AI issues. The purpose of this paper is to explore the potential for reconciliation between these two seemingly divergent factions. A reconciliation would enable them to expend less energy debating each other and more energy on activities that both agree are positive. In particular, if there are certain activities that simultaneously address issues associated with both near-term and long-term AI, then there is no need to engage in a near-term vs. long-term AI debate, because either position yields the same prescriptions. I will refer to the two factions as the futurists and the presentists. Each can be said to have a core claim: The futurist AI claim: Attention should go to the potential for radically transformative longterm AI. The presentist AI claim: Attention should go to existing and near-term AI. This division is a generalization in that some AI experts support both claims; there may even be some who support neither claim, though the rise of near-term AI applications makes it increasingly difficult for even non-experts to argue that AI is unworthy of attention. The futurist faction can be traced to the early days of AI. Early AI researchers such as I.J. Good and Marvin Minsky predicted that AI would soon be capable of human-level thinking. Foreshadowing contemporary futurist concerns,  Good (1965)  warned of self-improving ultraintelligent machines that would undergo "intelligence explosion" in which "the intelligence of man would be left far behind", forming "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control" (p.33). When the early predictions failed to pan out, the field entered into a more muted phase known as AI winter. Here began the presentist faction. Most AI researchers shifted from trying to build human-level general AI to building narrow, domain-specific AI. Thus "mainstream AI" came to be associated with developing narrow, near-term AI and repudiating AI futurism. While a minority of researchers continued work on human-level general AI (e.g.,  Goertzel and Pennachin 2007 ) and a few high-profile futurists brought long-term AI some public attention (e.g.,  Kurzweil 2006) , most AI researchers worked on narrow, near-term AI (see e.g.  Bostrom 2014, p.18; Goertzel 2014, p.1) . Within the last few years, two parallel developments have raised the prominence of the futurist-presentist AI divide. One is a breakthrough in the performance of certain AI systems, in particular those involving "deep learning". Major applications of AI are rapidly appearing, from medical diagnosis to self-driving cars. This breakthrough has generated considerable interest in near-term AI while making dramatic long-term AI projections appear more plausible. The other development is a spike in attention going to long-term AI, sparked mainly by the publication of the book Superintelligence  (Bostrom 2014) , public comments by several major celebrities (e.g.,  Hackett 2016; Hern 2016) , and accompanying efforts by futurist researchers and activists (e.g., Future of Life Institute no date). The sober, philosophical tone of Superintelligence and the perceived credibility of some new outspoken futurists has brought the futurist perspective a burst of serious consideration. This has in turn sparked backlash from the presentist faction. The following, from an op-ed in The New York Times by Microsoft principal researcher Kate Crawford, is representative: According to some prominent voices in the tech world, artificial intelligence presents a looming existential threat to humanity: Warnings by luminaries like Elon Musk and Nick Bostrom about "the singularity" -when machines become smarter than humans -have attracted millions of dollars and spawned a multitude of conferences. But this hand-wringing is a distraction from the very real problems with artificial intelligence today, which may already be exacerbating inequality in the workplace, at home and in our legal and judicial systems. Sexism, racism and other forms of discrimination are being built into the machinelearning algorithms that underlie the technology behind many "intelligent" systems that shape how we are categorized and advertised to  (Crawford 2016) . Crawford emphasizes inequality/discrimination as the near-term AI issue to focus on. Other presentists have emphasized such issues as military applications (e.g.,  Arkin 2009) , safety issues with medical applications or self-driving cars, disruptions to labor markets ("technological unemployment"), or simply the technical capabilities of the AI itself. The unifying theme for this group is its insistence on a focus on current and near-term AI. With growing interest in both near-term and long-term AI, the futurist-presentist divide has intensified. In addition to  Crawford (2016) , the divide can also be found in: a series of events on AI in 2016 sponsored by the White House, in which "many of the speakers… emphasized the need to focus on short-term concerns over long-term concerns of artificial general intelligence"  (Conn 2016a) ; remarks by AI researcher Andrew Ng, arguing that worrying about future AI is as unimportant as worrying about "overpopulation on Mars"  (Garling 2015) ; and an article by AI researcher Oren Etzioni, suggesting that AI that would not be built within the next 25 years is not worth paying attention to  (Etzioni 2016) , which prompted a reply by AI researcher Stuart Russell and political scientist Allan Dafoe arguing AI built more than 25 years from now can still be worth attention  (Dafoe and Russell 2016) ; see also  Bostrom (2014, p.18) . This sort of futurist-presentist divide is not unique to AI. For example, Selin (2007) documents a similar divide within the field of nanotechnology, in which a "mainstream" majority of researchers advocate for attention to (and funding for) existing and near-term nanotechnology at the expense of potential long-term transformative nanotechnology. Similarly, one explanation for the low priority commonly given to global warming is the fact that the harmful impacts of climate change will accrue mainly in the future  (Weber 2006) . In recognition of the challenge of motivating action on long-term issues,  Baum (2015)  proposes to address a wide range of longterm catastrophic risks-including long-term AI, nanotechnology, and climate change-by leveraging interest in synergistic near-term co-benefits. The present paper follows in a similar vein as  Baum (2015) , though it is not intended as a critique or an advocacy of either the futurist or presentist AI factions. Instead, it seeks to pragmatically pursue reconciliation between the factions so that they may be able to pursue mutually agreeable activities. Three such activities are described in Section 4. Leading up to that, Section 2 explores the ideas at the root of the futurist and presentist perspectives, while Section 3 proposes a reconciliation based on a general concern for the societal impacts of AI.
 Root Perspectives This section attempts to articulate the perspectives at the root of the futurist and presentist factions. The descriptions derive from readings from, observations of, and interactions with the two factions. Root perspectives can help explain why the two factions disagree with each other and where reconciliation may be possible.
 AI Futurists Two distinct root perspectives can be found among AI futurists. One is centered on the intellectual goals of AI research, the other on the implications of a certain ethical perspective. The first perspective is rooted in an intellectual aspiration of building advanced AI that have general intelligence at a human or superhuman level. "General" in this context means that they can think across a wide range of domains, or "have a broad capability to self-adapt to changes in their goals or circumstances"  (Goertzel 2014, p.1) . Building such advanced AI has long been "the grand dream of artificial intelligence"  (Legg 2008, p.125 ), but it is only pursued by a small portion of AI researchers. These are the researchers who are willing to accept that their work may be of little immediate practicality in order to work towards a more grand future achievement. They seek "to promote the idea that intelligent machines, even super intelligent machines, is a topic that is both important and one that can be scientifically studied, even if just theoretically for now"  (Legg 2008, p.126) .  Goertzel (2014, p.3)  presents what it calls the core hypothesis of artificial general intelligence: The creation and study of synthetic intelligences with sufficiently broad (e.g. humanlevel) scope and strong generalization capability, is at bottom qualitatively different from the creation and study of synthetic intelligences with significantly narrower scope and weaker generalization capability. Thus, advanced future AI requires fundamentally different research than is needed for the narrower contemporary AI.  Goertzel (2014)  maintains that this hypothesis is widely held among artificial general intelligence researchers. Thus, there is a community of researchers dedicated specifically to developing advanced future AI.  (Goertzel 2014 also surveys this research.)  The second AI futurist perspective starts with an ethical concern for long-term effects and pairs this with a belief that long-term AI can have major long-term effects. The ethical concern for long-term effects has rigorous philosophical justification, though as with any ethical position, it is not universally held. It follows directly from a principle of equality: if all human lives (or whatever else we might care about) should be valued equally, then this (arguably) includes future lives. Furthermore, there could be many more future lives than present lives. Thus, philosophers and economists have long recognized the importance of actions that can have ongoing benefits to future generations, such as economic saving  (Ramsey 1928 ) and survival of the population  (Koopmans 1974) . This same ethical perspective has been articulated more recently by some in the AI futurist faction (e.g.,  Bostrom 2003) . The perspective prompts people to seek out issues and actions that could have significant long-term impact; the "merely near-term" is seen as trivial in comparison. This big-picture, long-term perspective explains why the futurist AI faction disproportionately attracts philosophers (e.g., Bostrom 2014; Price 2013) and cosmologists (e.g.,  Hawking et al. 2014) , and why they give their organizations names like Future of Humanity Institute and Future of Life Institute. The long-term ethical perspective is not fundamentally about AI. Instead, it is a guide for all human action; any role for AI is circumstantial. It relates to AI via the belief that certain AI could have major long-term effects, sending human civilization on fundamentally different trajectories. Only AI that could be so transformative is worth paying attention to; the rest is trivial detail. Thus, the prospect of future transformative AI is seen as important, and indeed is important even if there is only a small probability of it being built and even if it would be built decades or centuries from now. As long as there is some nonzero probability that future AI could, say, take over the world and kill everyone  (Bostrom 2014)  or dominate the global economy while remaking the future of civilization  (Hanson 2016) , then it is worth paying attention to. Indeed, such transformative long-term AI competes for attention not with near-term AI but with other potentially transformative issues such as global warming, nuclear war, and long-term nanotechnology. The long-term ethical perspective can seem harsh in its relentless prioritization of long-term issues. For example,  Crawford (2016)  laments racial bias in AI evaluation of criminals in the United States, resulting in harsher prison sentences for African-Americans than for Americans of European descent. 1 Racial bias in law enforcement would seem to be an important issue, and indeed, at the time of this writing, it is at the very top of the public agenda in the United States. However, the attention is on racial bias as it affects people today, not on how it affects long-term outcomes for humanity. Indeed, it is possible that present cases of racial bias will have limited long-term effect, especially relative to a possible future transformative AI. The same holds for most, if not all, other near-term AI issues. Ironically, it is the same ethical principle of equality that underlies opposition to racial bias that also motivates the ignoring of racial bias in favor of long-term issues.
 AI Presentists In contrast, AI presentists are motivated by a desire to work on contemporary and near-term activities and issues. This holds for presentist AI researchers as well as those focused on societal issues raised by AI. Most AI research is presentist. The focus on contemporary and near-term AI systems is rooted in a practical, down-to-Earth perspective that seeks to work on actual systems and avoid speculation about future possibilities, especially when those possibilities seem strange and fantastic. Thus, for example, AI researcher Andrew Ng dismisses concerns about advanced future AI because "the future is so uncertain" that it cannot be meaningfully considered, and instead attention should be on problems that are more immediate and less ambiguous  (Garling 2015) . Similarly, Nilsson describes how AI researchers shy away from work on advanced future AI because they worry it would "risk losing our respectability"  (Nilsson 2010, p.399) . This near-term focus of AI research is not unusual. Looking across all fields of science and engineering, one finds few people motivated by long-term effects for humanity or other considerations deriving from theoretical philosophy. Indeed, they often believe ethical considerations to be fundamentally outside their professional domain, so much that they "accept radical restrictions on having consequential opinions about what ought to be done"  (Shapin 2010, p.388; emphasis original) . This can be seen, for example, in the United States National Science Foundation's two criteria for evaluating funding proposals: intellectual merit and broader impacts. Intellectual merit is what scientists and engineers consider intrinsically important; it is what wins grants and publications in top journals. Broader impacts is the societal significance of the research, which is often considered less important (e.g.,  Schienke et al. 2009) . AI researchers generally focus on near-term AI because that is what they consider to have more intellectual merit. Long-term AI is speculative and untestable, which places it outside the bounds of normal science and engineering. The potential for long-term AI to have larger societal impacts is not a factor. From this perspective, even talking about the long-term impacts can be counterproductive, leading to concerns about respectability, as noted above, and to boom-bust cycles of hype and disillusionment, as has been seen over the years of AI "summer" and "winter". Hence presentists can resent the mere mention of long-term AI. The norm of intellectual merit further explains why some AI researchers prefer to focus on the capabilities of the AI itself and not on societal issues. 2 Coming from computer science or related backgrounds, they view their role as one of advancing the science and technology of AI; attending to any related societal issues is, if anything someone else's job. However, many AI researchers are attentive to societal issues associated with AI.  Crawford (2016)  is not a rare exception.  3  When presentists do turn to societal issues, they turn to those that relate to existing and nearterm AI. This focus is reinforced by the fact that the general public and political leadership also tend to emphasize near-term issues. Hence even seemingly major long-term issues like global warming commonly get drowned out by near-term issues like economic growth  (Scruggs and Benegal 2012) . And so, when presentist AI researchers seek to discuss societal issues related to their AI, they find a welcome audience in social policy circles. This is seen, for example, in the heavy focus on near-term issues in the recent White House Symposia on AI Research  (Conn 2016a) .
 Reconciliation Despite the divergent perspectives between the presentists and futures, as well as the sometimes bitter debate between them, the two factions share a common interest in AI and belief of its importance. This commonality may seem banal, but it nonetheless distinguishes them from the large population that gives AI little attention or none at all. Thus, for starters, the two factions can at least unite in promoting AI as a locus of attention. But there can be much more than that. There are significant parts of both factions that are concerned about the societal impacts of AI. Their concerns may come from different places, the futurists starting with concern about a certain type of societal impact and then applying this to AI, and the presentists starting with an interest in a certain type of AI and then applying this to societal issues. The fact that they work in opposite directions explains their divergent viewpoints and AI emphases. But they can at least agree that the societal impacts of AI are important, worth paying attention to, and worth trying to improve, even while they disagree on which impacts to focus on. Meanwhile, there are those in both factions who are not so concerned about societal impacts, who instead prefer to focus on the AI itself. Thus, there is potential for a realignment of AI factions from futurist vs. presentist to those who are concerned about societal impacts vs. those who are not. This would seem to be the crucial distinction: between those who wish to pursue AI for its own sake or for narrow conceptions of intellectual merit, on the one hand, and on the other hand those who wish to pursue AI for the benefit of society. I will call these realigned factions the intellectualist faction and the societalist faction, rooted in the following core claims: The intellectualist AI claim: AI should be developed for its own sake, i.e. for its intellectual merit. The societalist AI claim: AI should be developed for its impacts for society. It should be clarified that societalists may often emphasize intellectual aspects of AI en route to pursuing societal benefits. This is to say that belonging to the societalist faction does not preclude one from pursuing intellectually advanced AI. There can be many opportunities to be a "good" AI researcher according to traditional science and engineering notions of intellectual merit while still working towards societal benefits. Another important clarification is that, among societalists, there will always be disagreements about which societal impacts to focus on; this is normal ethical and political debate. And just as conventional politics-the art of the possible-seeks areas of aligned interests, quid pro quo, and compromise, so too can those who disagree about the societal impacts of AI. This realignment has a big advantage in that it enables focus on promoting attention to societal impacts in general. It is straightforward to argue that presentists and futurists alike should be concerned about societal impacts. AI researchers, like other scientists and engineers or any other people, are not above the law or above morality. It is everyone's responsibility to help society.  4  The realignment has further advantages in the synergistic opportunities to be found between presentist and futurist societalists.
 Opportunities Within a realigned societalist faction, the best opportunities will generally be those in which the interests of presentists and futurists are aligned. Such opportunities require no compromise or other sacrifice. Several such opportunities are apparent.
 Social Norms A first opportunity is to encourage more AI researchers to care about the societal impacts of their work. This assumes-quite reasonably, one would think-that the societal impacts of AI tend to be better when more AI researchers aim to achieve better societal impacts. In other words, the pro-societal efforts of AI researchers tend to be productive, not counterproductive. While it is possible to persuade AI researchers one at a time to care about societal impacts, it will often be more effective to persuade many of them all together. This can be achieved by establishing societalist social norms among AI communities, i.e. by making it normal for AI researchers to seek to benefit society through their work. Societalist social norms are seen, for example, in AI researcher Stuart Russell's call for the AI field to switch from a norm of "building pure intelligence for its own sake, regardless of the associated objectives and their consequences" to a norm of not just caring about societal impacts, but also making that be "how practitioners define what they do"  (Bohannon 2015:252) . The advantage of advancing societalist social norms is that it creates more AI researchers available to work on pro-societal designs, policies, etc., and likewise fewer people pushing back against such activities. An expanded societalist faction should benefit both near-term and longterm societalist agendas: a rising tide lifts all boats. Given that AI research communities tend to focus on near-term AI, it may be advantageous to emphasize norms associated with the societal impacts of near-term AI. In practice, this means engaging AI researchers on such issues as inequality/discrimination (as in Crawford 2016), military applications (as in  Arkin 2009) , the safety of medical AI or self-driving cars, etc. It is relatively easy to sell the importance of these issues because either they already are important issues or they clearly will be in the near future, and because they do not depend on the arrival of any speculative future forms of AI but instead are rooted in existing and near-term AI. Social norms surrounding the societal impacts of near-term AI can be of indirect benefit to the societal impacts of long-term AI. One mechanism for this is the durability of norms. What is now long-term AI may one day become near-term AI. If societalist norms remain intact, then the near-term societalists would switch over to what is now long-term AI. There is reason to believe that societalist norms may be so durable. For comparison, norms against slavery or racism or sexism have proved durable; 5 within academia, the norm in favor of interdisciplinary research has gradually advanced for many years. If AI societalist norms could be similarly durable, they would help with what is now near-term and long-term AI. A second mechanism is for societalist norms to prompt AI researchers to become more interested in long-term impacts. This could happen via attention to the ethics underlying which societal impacts one might care about. Ethical reflection is a natural fit for societalist AI, i.e. societalists are likely to engage in ethical reflection. That reflection could include the ethics of future generations, which is precisely how many existing futurists came to be futurists. Therefore, a case can be made for promoting near-term societalist norms among AI communities. Because the issues are near-term, they are more likely to attract interest from AI communities. This increases the size of the near-term societalist faction. Then, some portion of these societalists may come to appreciate the ethical argument for caring about long-term impacts. This increases the size of the long-term societalist faction. Furthermore, the expanded societalist faction may remain in place when the long-term gets nearer. So, this is a win-win for both near-term and long-term societalists.
 Technical Research Societalist social norms expand the population of AI experts who can pursue the lines of technical AI research that benefits society. Some technical research may pertain only to specific AI applications and is thus only of interest to narrow presentist or futurist concerns. However, there is also some technical research that is of wide relevance to both near-term and long-term AI. Progress on this research is another win-win for near-term and long-term societalists. An example of this sort of "timeless" technical research can be found in  Amodei et al. (2016) . This paper describes a range of technical problems oriented towards avoiding unintended harms from AI systems. The paper is expressed in terms of unintended harms from near-term AI. However, in an interview, Amodei describes how he sees the technical problems as being relevant for near-term and long-term AI alike, such that the near-term/long-term distinction is unnecessary  (Conn 2016b) . Thus, all societalists could support this technical research agenda, regardless of whether they are ultimately concerned about near-term or long-term impacts. Indeed, in the same interview, Amodei further notes that his paper has received broad praise, including from people who fit into both the presentist and futurist factions  (Conn 2016b) .
 Policy Given the general orientation of policy communities towards near-term issues, it will typically be easier for policy to address issues associated with near-term AI. However, there are at least two ways that AI policy can concurrently support both near-term and long-term AI, making for a third win-win for near-term and long-term societalists. First, the process of developing near-term AI policy can create processes and competencies of relevance for long-term AI. Already, near-term AI policy issues have prompted legal scholars to familiarize themselves with some technical details of  AI (e.g., Calo 2011; Funkhouser 2013; Hammond 2015) . Such scholarship, and accompanying policy discussions, prepares policy communities for addressing long-term AI.  6  Near-term AI policy will also create demand for people with AI backgrounds working in policy positions. Such in-house AI expertise can help ensure that AI policy matches actual issues associated with AI technology and does not inadvertently restrict beneficial AI or enable harmful AI. Good models can be found, for example, in the Science & Technology Policy Fellowships offered by the American Association for the Advancement of Science and the IEEE-USA Government Fellowships offered by the Institute of Electrical and Electronics Engineers. These are both temporary fellowships; there should also be AI experts in permanent government positions. As long as the private sector offers abundant and lucrative employment for AI experts, it may be difficult to attract them to government positions. This challenge could be overcome via sufficiently strong societalist norms, which could motivate some societalists to forgo salary and pursue government positions simply because it is the right thing to do for society. Second, some specific policy measures may improve outcomes for both near-term and longterm AI. This may seem counterintuitive, given the rapidly changing nature of AI. Indeed, a common concern about policy for emerging technologies like AI is that policy will fail to keep up with changes in the technology. In response to this concern,  Moses (2007)  proposes to phrase policy in more general terms, such as "choose the AI design that is more beneficial to society"; such policy could remain relevant even if AI technology undergoes major changes. Other policies of enduring relevance can be policies that establish institutions for AI governance, including policies to establish support for pro-societal AI efforts.
 Conclusion The current division between AI futurists and presentists-i.e. those concerned mainly about long-term AI vs. those concerned mainly about near-term AI-is not the best focus of attention. Instead of bickering over which AI impacts are most important, both groups would be wise to focus on addressing the AI impacts in general. There are ample opportunities to concurrently address both near-term and long-term AI impacts, including by establishing societalist social norms within AI communities, by advancing technical research that improves societal outcomes for all types of AI, and by advancing public policy that improves the governance of all types of AI. Making progress on these three fronts will take effort, which is precisely why those who worry about AI impacts should not drain their energy on internal disputes. The Introduction to this paper quoted a passage from  Crawford (2016)  that argues for attention to near-term inequality/discrimination issues instead of to long-term threats. The passage is a representative example of the presentist vs. futurist dispute. However, in light of this paper's proposed reconciliation, the passage could be rewritten to instead take on the societalist vs. intellectualist dispute: According to some prominent voices in the tech world, artificial intelligence presents a major intellectual achievement, a triumph of science and technology. They say this achievement is to be celebrated, full stop. Indeed, research into the science of artificial intelligence has attracted millions of dollars and spawned a multitude of conferences. But this intellectual triumphalism is a distraction from the problems that artificial intelligence creates for society. It may already be exacerbating inequality in the workplace, at home and in our legal and judicial systems. It is also creating safety challenges for medical products and self-driving cars, as well as profound ethical dilemmas in its military applications. Looking further into the future, there may be some chance that artificial intelligence presents a looming existential threat to humanity. These are serious issues that demand serious attention from society as a whole and especially from the technologists whose work is at the heart of these issues. Expositions along these lines promise to reconcile the differences between presentists and futurists and instead shift attention to where it is desperately needed: on the societal issues posed by AI and on the positive steps can be taken to address these issues. 			 The racial bias Crawford describes comes from the investigative journalism of Angwin et al. (2016) .2  One can argue that the AI itself is situated within society, and thus that AI researchers inevitably work on societal issues, even when they believe that they are working only on the AI itself. However, for present purposes, what matters is that the AI researchers believe that they are focusing on the AI itself and not on societal issues, even if it can be argued that they are inadvertently working on societal issues.
			 As just one of many other examples, see Arkin (2009)  on societal issues associated with military robotics.
			 Again, as with any ethical position, the claim that people should help society is not universally held. A broader defense of this claim is beyond the scope of this paper.
			 The world still has slavery and racism and sexism, but not as much as it once did. For example, while the United States continues to grapple with a variety of racial biases, it has become unthinkable to support the "separate but equal" racial segregation of the former "Jim Crow" laws.6  There has also been some legal scholarship focused on long-termAI (e.g., McGinnis 2010; Wilson 2013) , but this is a relatively small minority of AI legal scholarship.


Abstract: We investigate synchronization in a network of continuous-time dynamical systems with smallworld connections. The small-world network is obtained by randomly adding a small fraction of connection in an originally nearest-neighbor coupled network. We show that, for any given coupling strength and a sufficiently large number of cells, the small-world dynamical network will synchronize, even if the original nearest-neighbor coupled network cannot achieve synchronization under the same condition.

SYNCHRONIZATION IN SMALL-WORLD DYNAMICAL NETWORKS

Introduction Collective motions of coupled dynamical networks are of significant interest in many fields of science and technology. In particular, synchronization in networks of coupled chaotic dynamical systems has received a great deal of attention in recent years. Most of the existing work on synchronization of coupled networks assumes that the coupling configuration is completely regular (see e.g.  [Heagy et al., 1994; Wu & Chua, 1995] ), while a few studies address the issue of synchronization in randomly coupled networks  [Gade, 1996; Manrubia & Mikhailov, 1999] . However, many biological, technological and social networks are neither completely regular nor completely random. To interpolate between these two extremes,  Watts and Strogatz [1998]  introduced the interesting concept of small-world networks. The so-called small-world networks have intermediate connectivity properties but exhibit a high degree of clustering as in the regular networks and a small average distance between vertices as in the random networks. They also found that the small-world networks of coupled phase oscillators can synchronize almost as readily as the globally coupled networks, despite the fact that they have much fewer edges  [Watts, 1999] . For a review of recent works on small-world networks, see  [Newman, 2000] . More recently,  Gade and Hu [2000]  explored the stability of synchronous chaos in coupled map lattices with small-world connectivity and found that in this case synchronous chaos is possible even in the thermodynamic limit.  Lago-Fernandez et al. [2000]  also investigated the fast response and temporal coherent oscillations in a small-world network of Hodgkin-Huxley neurons. In this study, we consider synchronization in a network of linearly coupled identical continuoustime dynamical systems. As shown by  Wu [1995] , for any given number of cells, strong enough mutual diffusive coupling will result in synchronization of the cells. Two commonly studied coupling configurations are the so-called easiest-toimplement nearest-neighbor coupling and the most difficult-to-implement global coupling. It has been shown that for any given coupling strength, if the number of cells is large enough, the globally coupled network will eventually synchronize, while the nearest-neighbor coupled network cannot achieve such synchronization under the same condition. This observation naturally poses the following question: for a nearest-neighbor coupled network with a sufficiently large number of cells and with an arbitrary coupling strength, is it possible to achieve synchronization of the network by a small modification of the nearest-neighbor coupling configuration, for example, by adding a small fraction of connection between some different pairs of cells? In this paper we provide a positive answer to this question based on the small-world network models.
 Preliminaries We consider a network of N identical cells, linearly coupled through the first state variable of each cell, with each cell being an n-dimensional dynamical subsystem. The state equations of the entire network are ẋi1 = f 1 (x i ) + c N j=1 a ij x j1 ẋi2 = f 2 (x i ) . . . ẋin = f n (x i ) i = 1, 2, . . . , N (1) where x i = (x i1 , x i2 , . . . , x in ) ∈ R n are the state variables of cell i, f i (0) = 0, c > 0 represents the coupling strength, and A = (a ij ) N ×N is the coupling matrix. In this paper, we only consider symmetric and diffusive coupling. In particular, we assume that (i) A is a symmetric and irreducible matrix. (ii) The off-diagonal elements, a ij (i = j) of A, are either 1 or 0 (when a connection between cell i and cell j is absent). (iii) The elements of A satisfy a ii = − N j=1 j =i a ij , i = 1, 2, . . . , N (2) The above conditions imply that one eigenvalue of A is zero, with multiplicity 1, and all the other eigenvalues of A are strictly negative. Given the dynamics of an isolated cell and the coupling strength, stability of the synchronization state of the network can be characterized by those nonzero eigenvalues of the coupling matrix. A typical result states that the network will synchronize if these eigenvalues are negative enough  [Wu & Chua, 1995] . Lemma 1. Consider network (1). Let λ 1 be the largest nonzero eigenvalue of the coupling matrix A of the network. The synchronization state of network (1) defined by x 1 = x 2 = • • • = x n is asymp- totically stable, if λ 1 ≤ − T c (3) where c > 0 is the coupling strength of the network and T > 0 is a positive constant such that zero is an exponentially stable point of the n-dimensional system ż1 = f 1 (z) − T z 1 ż2 = f 2 (z) . . . żn = f n (z) (4) Note that system (4) is actually a single cell model with self-feedback −T z 1 . Condition (3) means that the entire network will synchronize provided that λ 1 is negative enough, e.g. it is sufficient to be less than −T/c, where T is a constant so that the self-feedback term −T z 1 could stabilize an isolated cell. As mentioned above, two commonly studied coupling configurations are the nearest-neighbor coupling and the global coupling ones. Experimentally, the nearest-neighbor coupling is perhaps the easiest one to implement and, on the contrary, the global coupling is the most expensive one to implement. The nearest-neighbor coupling configuration consists of cells arranged in a ring and coupled to the nearest neighbors. The corresponding coupling matrix is A nc =          −2 1 1 1 −2 1 . . . . . . . . . 1 −2 1 1 1 −2          (5) The eigenvalues of A nc are −4 sin 2 kπ N , k = 0, 1, . . . , N − 1 (6) Therefore, according to Lemma 1, the nearestneighbor coupled network will asymptotically synchronize if 4 sin 2 π N ≥ T c (7) The global coupled configuration means that any two different cells are connected directly. The corresponding coupling matrix is A gc =          −N + 1 1 1 • • • 1 1 −N + 1 1 • • • 1 . . . . . . . . . . . . . . . 1 1 1 • • • 1 1 1 1 • • • −N + 1          (8) Matrix A gc has a single eigenvalue at 0 and all the others equal to −N . Hence, Lemma 1 implies that this network will asymptotically synchronize if N ≥ T c (9) In summary, for any given coupling strength c > 0, the globally coupled network can synchronize as long as the number of cells N is large enough. On the other hand, since sin(π/N ) decreases to zero as N increases, relation (7) cannot hold for sufficiently large N . Simulations also that the nearestneighbor coupled network cannot synchronize if the number of cells is sufficiently large. Thus, we have seen a trade-off between these two situations.
 Synchronization in Small-World Networks Aiming to describe a transition from a regular network to a random network,  Watts and Strogatz [1998]  introduced an interesting model, now referred to as the small-world (SW) network. The original SW model can be described as follows. Take a onedimensional lattice of N vertices arranged in a ring with connections between only nearest neighbors. We "rewire" each connection with some probability, p. Rewiring in this context means shifting one end of the connection to a new vertex chosen at random from the whole lattice, with the constraint that no two different vertices can have more than one connection in between, and no vertex can have a connection with itself. Note, however, that there is a possibility for the SW model to be broken into unconnected clusters. This problem can be circumvented by a slight modification of the SW model, suggested by  Newman and Watts [1999] , which is referred to as the NW model hereafter. In the NW model, we do not break any connection between any two nearest neighbors. We add with probability p a connection between each other pair of vertices. Likewise, we do not allow a vertex to be coupled to another vertex more than once, or coupling of a vertex with itself. For p = 0, it reduces to the originally nearest-neighbor coupled system; for p = 1, it becomes a globally coupled system. In this paper, we are interested in the NW model with 0 < p < 1. From a coupling-matrix point of view, network (1) with small-world connections amount to that, in the nearest-neighbor coupling matrix A nc , if a ij = 0, we set a ij = a ji = 1 with probability p. Then, we recompute the diagonal elements according to formula (2). We denote the new small-world coupling matrix by A ns (p, N ) and let λ 1ns (p, N ) be its largest nonzero eigenvalue. According to Lemma 1, if λ 1ns (p, N ) ≤ − T c (10) then the corresponding network with small-world connections will synchronize. Figures  1 and 2  show the numerical values of λ 1ns (p, N ) as a function of the probability p and the number of cells N . In these figures, for each pair of values of p and N , λ 1ns (p, N ) is obtained by averaging the results of 20 runs. It can be seen that (i) For any given value of N , λ 1ns (p, N ) decreases to −N as p increases from 0 to 1. (ii) For any given value of p ∈ (0, 1], λ 1ns (p, N ) decreases to −∞ as N increases to +∞. The above results imply that, for any given coupling strength c > 0, we have (i) For any given N > T/c, there exists a critical value p so that if p ≤ p ≤ 1, then the smallworld connected network will synchronize. (ii) For any given p ∈ (0, 1], there exists a critical value N so that if N ≥ N , then the small-world connected network will synchronize.
 Synchronization in a Network of Small-World Coupled Chua's Circuits As an example, we now study synchronization in a network of small-world connected Chua's circuits. In the dimensionless form, a single Chua's circuit is described by  [Chua et al., 1993] :    ẋ1 ẋ2 ẋ3    =    α(x 2 − x 1 + f (x 1 )) x 1 − x 2 + x 3 −βx 2 − γx 3    (11) where f (•) is a piecewise-linear function, f (x 1 ) =      −bx 1 − a + b x 1 > 1 −ax 1 |x 1 | ≤ 1 −bx 1 + a − b x 1 < −1 (12) in which α > 0, β > 0, γ > 0, and a < b < 0. The state equations of the entire network are    ẋi1 ẋi2 ẋi3    =        α(x i2 − x i1 + f (x i1 )) + c N j=1 a ij x j1 x i1 − x i2 + x i3 −βx i2 − γx i3        , i = 1, 2, . . . , N . (13) For this network to synchronize, according to Lemma 1, we may take T = −α. In simulations, the system parameters are chosen to be  For this set of parameters, Chua's circuit (11) has a chaotic attractor, as shown in Fig.  3 . The nearestneighbor coupled Chua's network cannot synchronize for N > 6. According to Lemma 1, the smallworld network will synchronize if λ 1ns (p, N ) ≤ a c = −1.27 (15) Figure  4  shows the values of p and N which can achieve network synchronization. For example, for N = 100, 150 and 200, synchronization of the small-world connected network can be achieved, for p > 0.0366, p > 0.0257 and p > 0.021, respectively.
 Conclusions Starting with a nearest-neighbor coupled dynamical network, we can construct a small-world dynamical network by adding with probability p a connection between each of the other pair of cells. We found that, for any given coupling strength and a sufficiently large number of cells, synchronization in a network of linearly small-world coupled continuoustime dynamical systems can be achieved with a small value of p. In other words, the ability of achieving synchronization in an originally nearestneighbor coupled system can be greatly enhanced by simply adding a small fraction of new connection, revealing an advantage of small-world network for chaos synchronization. Fig. 1 .Fig. 2 . 12 Fig. 1. Numerical values of λ1ns(p, N) as a function of the probability p: (a) N = 200; (b) N = 500.
 α Fig. 3. Chaotic attractor of Chua's circuit (11), with parameters given in (14).
 Fig. 4 . 4 Fig. 4. Values of p and N achieving synchronization in the small-world network of Chua's circuits.


Abstract: Hypothetical "value learning" AIs learn human values and then try to act according to those values. The design of such AIs, however, is hampered by the fact that there exists no satisfactory definition of what exactly human values are. After arguing that the standard concept of preference is insufficient as a definition, I draw on reinforcement learning theory, emotion research, and moral psychology to offer an alternative definition. In this definition, human values are conceptualized as mental representations that encode the brain's value function (in the reinforcement learning sense) by being imbued with a context-sensitive affective gloss. I finish with a discussion of the implications that this hypothesis has on the design of value learners.

Defining Human Values for Value Learners

Introduction 1 The value learning problem  (Dewey 2011 , Soares 2014  is the challenge of building AI systems which can first learn human values and then to act in accordance to them. Approaches such as inverse reinforcement learning  (Ng & S. Russell 2000)  have been suggested for this problem (S.  Russell 2015 , Sezener 2015 , as have more elaborate ones such as attempting to extrapolate the future of humanity's moral development  (Yudkowsky 2004 , Tarleton 2010 , Muehlhauser & Helm 2012 . However, none of these proposals have yet offered a satisfactory definition of what exactly human values are, which is a serious shortcoming for any attempts to build an AI system that was intended to learn those values. This paper builds on a combination of research into moral psychology, the psychology of emotion, and reinforcement learning theory to offer a preliminary definition of human values, and how that definition might be used to design a value learning agent. I begin with an argument for the standard concept of preference being insufficient as a definition of value in section 2. Section 3 introduces theoretical background from the field of reinforcement learning and particularly evolutionary reinforcement learning. The background is used in section 4 to offer a preliminary definition of human values as mental representations which encode the brain's value function (in the reinforcement learning sense, as discussed below) by being imbued with affect. Section 5 elaborates on why affect might be a reasonable candidate for the brain's way of encoding a value function, and section 6 discusses the connections between emotions, affect, and values. Section 7 discusses the relation of affect and moral judgment in light of the social intuitionist model of morality, and section 8 talks about how this concept of human values could be used for designing value learning agents. Sections 9 and 10 conclude by evaluating the model and comparing it to alternatives.
 The standard concept of preference is insufficient as a definition of value The closest existing concept that formalizes something akin to human value is the concept of a utility function, which is widely used in economics and decision theory. Possibly its most well-known problem as a model of value is the empirical finding that humans violate the axioms of utility theory and thus do not have consistent utility functions  (Tversky and Kahneman 1981) . However, this is far from being the most serious problem. The von Neumann-Morgenstern utility theorem  (von Neumann & Morgenstern 1953)  sets up utility functions via preference orderings: of options A and B, either A is preferred to B, B is preferred to A, or both are equally preferred. Essentially, a "preference" is defined as a function that, given the state of the agent and the state of the world in general, outputs an agent's decision between two or more choices. A strength of this definition is that it allows treating preferences as black boxes. This has been of great use in economics, as it allows constructing models of behavior The Workshops of the Thirtieth AAAI Conference on Artificial Intelligence AI, Ethics, and Society: Technical Report WS-16-02 based only on observed preferences, without needing to know the reasons for those preferences. At the same time, ignoring everything that happens inside the preference function is also a weakness for the definition. Preferences are essentially considered atomic units with no internal structure. This leads to a number of problems in trying to use them as a definition of human values, including the below. The utility function model of value has difficulty dealing with internal conflicts and higher-order preferences. A drug addict may desire a drug, while also desiring that he not desire it  (Frankfurt 1971) . "Less Is More" is a measure of executive function in which children may point either to a tray with five treats or to a tray with two treats, while knowing that they will get the treats from the tray which they didn't point at. Three-year old children frequently fail this task and point at the tray with more treats, despite understanding that this will give them fewer things that they want  (Carlson et al. 2005) . Although the researchers did not report the children's reaction to their repeated failure, it seems safe to presume that they were not particularly happy, nor would they have liked to have their preference modeled as preferring fewer treats. The utility function model of value ignores the person's internal experience. Although "wanting" and "liking" are frequently thought to be the same thing, the two have distinct neural processes: "[l]iking corresponds closely to the concept of palatability; wanting, by comparison, corresponds more closely to appetite or craving"  (Berridge 1996) . Different interventions may suppress wanting without affecting liking, and vice versa. Intuitively, it seems like behaviors which we both "like" and "want" should be more important than behaviors that we only "want". The utility function model of value does not model changing values. As a black box mechanism, classical preference has no model of changing values, preventing us from extrapolating possible development of values. The utility function model of value does not give a way to generalize from our existing values to new ones. Technological and social change frequently restructures the way that the world works, forcing us to reconsider our attitude towards the changed circumstances. As a historical example (Lessig 2004), American law traditionally held that a landowner did not only control his land but also everything above it, to "an indefinite extent, upwards". Upon the invention of this airplane, this raised the question: could landowners forbid airplanes from flying over their land, or was the ownership of the land limited to some specific height, above which the landowners had no control? The US Congress chose to the latter, designating the airways as public, with the Supreme Court choosing to uphold the decision in a 1946 case. Justice Douglas wrote in the court's majority that The air is a public highway, as Congress has declared. Were that not true, every transcontinental flight would subject the operator to countless trespass suits. Common sense revolts at the idea. By the decision of Congress and the Supreme Court, the concept of landownership was redefined to only extend a limited, and not an indefinite, amount upwards. Intuitively, one might think that this decision was made because the redefined concept did not substantially weaken the position of landowners, while allowing for entirely new possibilities for travel. However, a black-box approach to value, which does not reveal the reasons underlying preferences such as "landownership should extend indefinitely upwards", would be incapable of making such a judgment.
 Evolutionary reinforcement learning A good theory of human psychology, including human value, requires an understanding of the evolutionary functions of the psychological phenomena  (Tooby & Cosmides 1995 ). Before we can develop a good model of what human values are, we need to develop an understanding of their computational role in the kinds of tasks that human brain has needed to perform. A defining characteristic of human thought is the ability to develop solutions to novel problems in novel environments. Humans are capable of learning a wide variety of behaviors far beyond anything that evolution could have "preprogrammed" into them. Instead, they experience some events (such as tissue damage or hunger) as aversive and learn to avoid things that cause those events, while learning to pursue things that feel rewarding. The problem of learning a novel environment in order to maximize the amount of rewards is the reinforcement learning problem, which "explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment"  (Sutton & Barto 1998) , as opposed to merely considering some isolated subproblems. As the theory of reinforcement learning is the general answer to the question of how an agent should behave in an uncertain environment and learn from it, we should expect the design of both human and animal minds to be strongly shaped by the principles of reinforcement learning theory. Empirical evidence from a variety of fields, including behavioral trainers  (Pryor 1999) , studies on habit-formation  (Duhigg 2012)  as well as neuroscience  (Dayan 2011 ) supports this prediction. Standard reinforcement learning theory involves learning to maximize a reward signal which the agent can observe. However, evolution selects for traits with the highest inclusive fitness, an abstract measure of a trait's effect on the production and survival of direct and related offspring. As organisms cannot directly observe the effect of their actions on their lifetime fitness, they cannot maximize this value directly.  Singh et al. (2009 Singh et al. ( , 2010  expand reinforcement learning to cover the evolutionary case, and define an "optimal reward function" as follows. An agent A in an external environment e receives observations and takes actions. It has an internal environment which computes a state based on the observations from the environment. The agent tries to maximize a reward, which is also computed by the internal environment according to a reward function rA, drawn from the space of reward functions RA. Different agents carry out actions in various environments e drawn from a distribution of environments E. A specific agent A in environment e with reward function rA produces a history h. A fitness function F produces a scalar evaluation F(h) for all histories h. A reward function is optimal if it maximizes the expected fitness of the agent over the distribution of environments. This formalization mimics an evolutionary environment in that evolution selects for agents which best maximize their fitness, while agents cannot directly optimize for their own fitness as they are unaware of it. Agents can however have a reward function that rewards behaviors which increase the fitness of the agents. The optimal reward function is one which maximizes (in expectation) the fitness of any agents having it. Holding the intelligence of the agents constant, the closer an agent's reward function is to the optimal reward function, the higher their fitness will be. Evolution should thus be expected to select for reward functions that are closest to the optimal reward function. In other words, organisms should be expected to receive rewards for carrying out tasks which have been evolutionarily adaptive in the past.
 An initial definition of human value The theory of reinforcement learning distinguishes between reward and value. A value function assigns states of the world a scalar value based on the expectation of future rewards that one may obtain from that state, conditional on some policy of what one would do in each state. Intuitively, a state has a high value if one can reliably move from it to states with a high reward. For reinforcement learning to work effectively, it requires a way to identify states which should be classified as the same or similar, and be assigned the same or a similar value. We can now consider the relation between the need to identify similar states, and mental concepts. We should expect an evolutionarily successful organism to develop concepts that abstract over situations that are similar with regards to receiving a reward from the optimal reward function. Suppose that a certain action in state s1 gives the organism a reward, and that there are also states s2-s5 in which taking some specific action causes the organism to end up in s1. Then we should expect the organism to develop a common concept for being in the states s2-s5, and we should expect that concept to be "more similar" to the concept of being in state s1 than to the concept of being in some state that was many actions away. Empirical support for concepts being organized in this kind of a manner comes from possibly the most sophisticated general-purpose AI developed so far, DeepMind's deep reinforcement learning agent  (Mnih et al. 2015) . This agent managed to "achieve a level comparable to that of a professional human games tester across a set of 49 [Atari 2600] games, using the same algorithm, network architecture and hyperparameters". This agent developed an internal representation of the different game states of each game that it was playing. An investigation of the agent's representation for the game Space Invaders indicated that representations with similar values were mapped closer to each other in the representation space. Also, some game states which were visually dissimilar to each other, but had a similar value, were mapped to internal representations that were close to each other. Likewise, states that were visually similar but had a differing value were mapped away from each other. We could say that the agent learned a primitive concept space, where the relationships between the concepts (representing game states) depended on their value and the ease of moving from one game state to another. There is considerable disagreement on what exactly concepts are, and various theoreticians use the same term to refer to different things  (Machery 2010) . For the purposes of this paper, I am loosely defining a "concept" as points or regions within a conceptual space, with concepts having a hierarchical structure so that higherlevel concepts are at least partially defined in terms of lower-level ones. Similar assumptions are commonly made in psychology  (Gärdenfors 2004 ) and neuroscience  (Wessinger et al. 2001) . Additionally, this definition makes concepts remarkably similar to the representations built up in the machine learning subfield of deep learning. Deep learning models have demonstrated success in a wide range of tasks, including object recognition, speech recognition, signal processing, natural language processing and transfer learning  (Bengio 2012 , Schmidhuber 2014 . They work by building up an internal representation of a domain, where different concepts are arranged in a hierarchical structure, with more abstract concepts at the top. These ideas allow us to establish a preliminary definition of value in the "human value" sense. I suggest that human values are concepts which abstract over situations in which we've previously received rewards, making those concepts and the situations associated with them valued for their own sake. A further suggestion is that, as humans tend to naturally find various mental concepts to be associated with affect (the subjective experience of a feeling or emotion, experienced as either positive or negative), the value function might be at least partially encoded in the affect of the various concepts. In support of this possibility, I next turn to some of the research studying the role of affect in decision-making.
 Affect as a possible representation for the value function Affective evaluations of concepts seem to influence people's behavior. For instance,  Benthin et al. (1995)  found that the experienced affective feel of mental images associated with various health-related behaviors predicted the extent to which high schoolers engaged in those behaviors. Another study  (Peters & Slovic 1996 ) surveyed a representative sample of the US adult population. This study found that both the respondents' general worldview and their affective associations with nuclear power predicted the respondents' support for nuclear power independently of each other. This kind of a reliance on immediate affective responses to various options in guiding decision-making has been named the affect heuristic, and documented in a number of studies  (Slovic et al. 2007) . However, the dissociation between "wanting" and "liking"  (Berridge 1996)  suggests that the value function may not be completely contained in affective evaluations, as it is possible to "want" things without "liking" them, and vice versa. I am choosing to regardless mainly focus on the affective ("liking") component. This is due to the intuition that, in the context of looking for a target of value learning, the values that are truly important for us are those that involve a "liking" component, rather than the values with a "wanting" component without a "liking" component. The former seem closer to things that we like and enjoy doing, while the latter might be closer to behaviors such as undesired compulsions. I wish to emphasize, however, that this is only a preliminary conjecture and one which still needs further investigation. In order to be a good candidate for the representation of a value function, the affect of different concepts should vary based on contextual parameters such as the internal state of the organism, as (for example) a hungry and nonhungry state should yield different behaviors. Rats who taste intense salt usually both "dislike" and "unwant" it, but when they become salt-deprived, they will start both "wanting" and "liking" the salt, with the "wanting" expressing itself even before they have had the chance to taste the salt in the salt-deprived state and consciously realize that they now enjoy it  (Tindell et al. 2009) . Thus it seems that both the affective value and "wanting" of something can be recomputed based on context and the organism's own state, as would be expected for something encoding a value function. Similarly, a state such as fear causes shifts on our conceptual frames, such as in the example of a person who's outside alone at night starting to view their environment in terms of "dangerous" and "safe", and suddenly viewing some of their familiar and comfortable routes as aversive  (Cosmides & Tooby 2004 ). This is another indication of the affect values of different concepts being appropriately context-dependent. The negative or positive affect associated with a concept may also spread to other related concepts, again as one would expect from something encoding a value function. A person who is assaulted on a particular street may come to feel fear when thinking about walking on that street again. The need to be ready to confront one's fears and pains is also emphasized in some forms of therapy: if a person with a fear of snakes turns down invitations to go to a zoo out of a fear of seeing snakes there, they may eventually also become anxious about any situation in which they might be invited to a zoo, and then of any situation that might lead to those kinds of situations, and so on  (Hayes & Smith 2005) . Such a gradual spreading of the negative affect from the original source to related situations seems highly similar to a reinforcement learning agent which is updating its value function by propagating the value of a state to other states which precede it.
 Human values and emotions Human values are typically strongly related to emotional influences, so a theory which seeks to derive human values from a reinforcement learning framework also needs to integrate emotions with reinforcement learning. A major strand of emotion research involves appraisal theories  (Scherer 1999 , Roseman & Smith 2001 , Scherer 2009 , according to which emotional responses are the result of an individual's evaluations (appraisals) of various events and situations. For example, a feeling of sadness might be the result of an evaluation that something has been forever lost. The evaluations then trigger various responses that, ideally, orient the organism towards acting in a manner appropriate to the situation. After an evaluation suggests that something important has been forever lost, the resulting sadness may cause passivity and a disengagement from active goal pursuit, an appropriate response if there is nothing that could be done about the situation and attempts to pursue the goal would only lead to wasting resources  (Roseman & Smith 2001) . An important property of emotional appraisals is that different situations which might cause the same evaluation may not have any physical features in common with each other: Physically dissimilar events (such as the death of a parent and the birth of a child) may produce the same emotion (e.g. sadness) if they are appraised in similar ways (e.g. as involving the loss of something valued). An infinite number of situations can elicit the emotion because any situation that is appraised as specified will evoke the same emotion, including situations that have never before been encountered. Thus, the loss of one's first love or first cherished possession is likely to elicit sadness; and if people develop the ability to clone copies of themselves, a man who wants this capability but believes that he has lost it will feel sad.  (Roseman & Smith 2001)  In other words, emotional responses are a result of appraisals abstracting over situations which are similar on some specific property that has been evolutionarily important. As such, in addition to their direct psychological and physiological effects, they could also be seen as providing a reinforcement learning agent with information about which states are similar and should be treated as similar for learning purposes. Emotions are also associated with an affect dimension, with the conscious experience of an emotion often being theorized as being the integral blend of its affect (unpleasant-pleasant) and arousal (lethargic-energetic) dimensions (J.  Russell 2003) . Combining the above ideas, it is natural to suggest that since emotional appraisals identify evolutionarily important states, the optimal reward function for humans' environment of evolutionary adaptedness (EEA,  Tooby & Cosmides 1990 ) has involved positive rewards for emotions which reflect desirable states, and negative rewards for emotions which reflect undesirable states.  Marinier & Laird (2008)  experimented with implementing a reinforcement learning-driven agent with simple appraisal-based emotions in a toy environment. They found the agent with emotions to learn faster than a standard reinforcement learning agent, as the emotionequipped agent received frequent feedback of its progress from its appraisals and thus learned faster than the standard agent, which only received feedback when it reached its goal. Humans and many animals also enjoy exploration and play from a very early age, in a manner which cannot be explained by those exploratory behaviors having been reinforced by other rewards.  Singh et al. (2009)  set up a simulated environment in which agents could move about and take actions for the first half of their lifetimes, but could not yet carry out actions that would increase their fitness. In the second half of the agents' lifetimes, actions which increased their fitness became available. The authors found that the optimal reward function for this kind of an environment is one that rewards the agents for learning simple behaviors that can be performed during their "childhood", and which are prerequisites for the more complex fitness-increasing behaviors. Once the more complicated fitness-increasing behaviors become possible during the "adulthood" of the agents, agents with a reward function that has already taught them the simpler forms of the behavior will increase their fitness faster than agents that do not engage in "childhood play" and have to learn the whole behavioral chain from scratch. This and similar examples  (Singh et al. 2010 ) on the value of behaviors such as curiosity and play provide an explanation for why humans would find those behaviors rewarding for their own sake, even when the humans were modeled as reinforcement learners who did not necessarily receive any other rewards from their play.
 Human values and morality The discussion so far has suggested that human values are concepts that have come to be associated with rewards, and are thus imbued with a (context-sensitive) level of affect. However, I have said little about morality in particular. The social intuitionist model of moral psychology  (Haidt 2001)  proposes that moral judgment is "generally the result of quick, automatic evaluations (intuitions)". It can be contrasted to rationalist models, in which moral judgments are the results of careful moral reasoning.  Haidt (2001)  begins with a discussion of people's typical reactions to the following vignette: Julie and Mark are brother and sister. They are traveling together in France on summer vacation from college. One night they are staying alone in a cabin near the beach. They decide that it would be interesting and fun if they tried making love. At the very least it would be a new experience for each of them. Julie was already taking birth control pills, but Mark uses a condom too, just to be safe. They both enjoy making love, but they decide not to do it again. They keep that night as a special secret, which makes them feel even closer to each other. What do you think about that? Was it OK for them to make love?  Haidt (2001)  notes that most people will have an instant negative reaction to the vignette and say that what the siblings did was wrong  (Haidt et al. 2000 ). Yet the reasons that people offer for the act having been wrong are inconsistent with the story that was presented: for example, people might offer the possibility of birth defects from inbreeding, only to be reminded that the siblings were thorough in using birth control. This is used as an illustration of  Haidt's (2001)  claim that "moral reasoning is usually a post hoc construction, generated after a judgment has been reached". In particular, moral judgments are thought to be strongly driven by moral intuitions, which are defined as ...the sudden appearance in consciousness of a moral judgment, including an affective valence (good-bad, like-dislike), without any conscious awareness of having gone through steps of searching, weighing evidence, or inferring a conclusion. Moral intuition is therefore the psychological process that the Scottish philosophers talked about, a process akin to aesthetic judgment: One sees or hears about a social event and one instantly feels approval or disapproval.  (Haidt 2001)  I suggest that the social intuitionist model is highly compatible with my framework. Recall that I defined human values as concepts which have become strongly associated with positive or negative affect. Something like a moral intuition for brother-sibling incest being something abhorrent, could be explained if something like the hypothesized Westermarck Effect (Rantala & Marcinkowska 2011) made individuals find the concept of having sex with their siblings to be abhorrent and strongly laden with negative affect. Thus the concept of incest would instantly cause a negative reaction, leading to a moral judgment of condemnation. The social part of social intuitionist theory emphasizes the impact of culture and the social environment in shaping various moral intuitions.  Haidt (2001)  suggests at least three kinds of cultural processes which shape intuitions: 1. Selective loss of intuitions is the suggestion that people are from birth capable of developing many different kinds of intuitions, but that intuitions which are not emphasized by the prevailing culture gradually become weaker and less accessible. This is suggest to possibly be analogous to the process in which children lose the ability to distinguish between phonemes which are not distinguished in their native language. 2. Immersion in custom complexes. Various customs that are practiced in a culture are hypothesized to affect the implicit beliefs of people growing up in that culture. For example, the culture in Orissa, India structures spaces and objects by rules of purity and pollution. This involves rules such as dividing temples to zones of increasing purity, with foreigners and dogs being allowed near the entrance, bathed worshippers being allowed into the courtyard, and only the Brahmin priest being allowed into the inner sanctum. It is suggested that after a life of navigating such rules and practices, children internalize a way of thought that makes later intellectual concepts of sacredness, asceticism and transcendence feel natural and self-evident. It is interesting to note that this suggestion maps fits naturally into my suggested model of the role of concepts. If the function of concepts is to foster the right behavior in the right situations, then a person who is required by their culture to internalize a set of allowed, required, and disallowed behaviors in various high-or low-purity zones needs to develop a set of concepts which link the right behaviors together with the appropriate levels of purity. Once this conceptual network is in place, even if only in an implicit and unconscious level, new concepts which share a similar structure with the previously-learned one may feel easy and intuitive to develop. 3. Peer socialization. Many moral intuitions are learned from the culture in one's peer group; in particular, there might be evidence that immersion within a culture between the ages of 9 and 15 causes permanent internalization of the norms of that culture in a way that causes them to feel obvious and intuitive. Social intuitionist theory proposes that moral judgments involve a number of moral intuitions, but does not explicitly outline what those intuitions are or where they come from. Other theories building on social intuitionism, such as moral foundations theory  (Graham et al. 2012 , Haidt 2012 ) have proposed various foundations from which the intuitions are derived. For example, the care/harm foundation is hypothesized to have its origins in an evolutionary adaptation to care for one's offspring, and to motivate people to care for others and to help them avoid harm. While my model is not committed to any particular set of moral intuitions, theories such as moral foundations are broadly compatible with the model, offering an additional set of sources through which concepts may become imbued with either positive or negative affect.
 Building value learners In my framework, various sources of reward lead the brain to calculate an approximation of a value function, which then becomes expressed in the affect of various concepts. This preliminary definition of values seems to suggest ways to implement value learning in ways which avoid some of the problems associated with a more structure-free model of preferences. I have discussed some sources of reward, including classical physical events such as food or physical pain, the affective dimension in various emotional reactions, and moral intuitions. A further candidate for a source of reward might be something like self-determination theory, which posits the existence of the three universal human needs of competence, autonomy and relatedness  (Ryan & Deci 2000) . I do not expect this list to be comprehensive, and is rather intended as illustrative only. A value learning agent attempting to learn the values of a human might first map the sources of reward for that human. Given a suitable definition of sources of reward, discovering the various sources for any given individual seems like the kind of a task that might be plausibly outsourced to the AI system, without all of the sources needing to be exhaustively discovered by human researchers first. Having this information would assist the value learner in mapping the individual's values, in the form of a map of concepts and their associated affect values. Once the map had been obtained, it could be used to generate preference rankings of different outcomes for the individual and the world, using similar mechanisms as the human brain does when considering the appeal of different outcomes. When unsure about the desirability of some scenario, the value learner could attempt to model the amount of reward that the human would receive from living in the circumstances implied by that scenario, and use this to evaluate what value the human would then come to place on the scenario. One issue with this model is that there is no theoretical reason to expect that a person's values would necessarily imply consistent preference orderings, as people tend to have contradictory values. To resolve this issue, I turn back to the social intuitionist models of morality. While social intuitionism places stronger value on moral intuitions than rationalist models do, it must be emphasized that it incorporates, rather than rejects, the possibility of moral reasoning as well.  Haidt (2001)    writes:  ...people may sometimes engage in private moral reasoning for themselves, particularly when their initial intuitions conflict. Abortion may feel wrong to many people when they think about the fetus but right when their attention shifts to the woman. When competing intuitions are evenly matched, the judgment system becomes deadlocked  [...] . Under such circumstances one may go through repeated cycles of [...] using reasoning and intuition together to break the deadlock. That is, if one consciously examines a dilemma, focusing in turn on each party involved, various intuitions will be triggered [...], leading to various contradictory judgments  [...] . Reasoning can then be used to construct a case to support each judgment  [...] . If reasoning more successfully builds a case for one of the judgments than for the others, the judgment will begin to feel right and there will be less temptation (and ability) to consider additional points of view. [...] We use conscious reflection to mull over a problem until one side feels right. Then we stop. I suspect that to the extent that this model of human moral reasoning is correct, it could also be used by the value learner to predict how the individual being modeled might resolve any particular inconsistency. Because the value learner could potentially consider a far larger set of relevant intuitions at once, it could also help the individual in their moral growth by nudging them to consider particular scenarios they might otherwise not have considered. Drawing partial inspiration from Christiano's (2014) notion of approval-directed agents, this kind of an approach might also help avoid classical problems with seeking to maximize the fulfillment of an individual's preferences, such as the possibility of rewriting a person's mind to maximize the amount of reward the person obtained. This could be accomplished by applying the same criteria for evaluating potential outcomes, into evaluating the value learner's actions. Would the human's values approve the scenario where the value learner attempted to rewrite the human's brain? If the answer is no, that course of action would be prohibited. Depending on the exact path taken, it seems reasonable to assume that this kind of a moral reasoning process might arrive at several different conclusions. Conceivably, if an examination of different perspectives makes some side in a moral dilemma feel more right than the other, the intuitions that were on the losing side might become weakened as a result. If those intuitions also affected the outcome of some other moral dilemma, then the order in which the dilemmas were attended to might determine the conclusions that the individual reaches. Further history-dependence is introduced by the role of social effects, in which people affect the values of the people around them. This raises interesting possibilities in introducing flexibility to the value learner. If there is no single correct set of final values that an individual's or a society's values might converge to, the value learner might be allowed to nudge the individual towards such a set of values whose implied preference ordering on the world would be the easiest to fulfill. Depending on the individual's values, they might allow this kind of a nudging, or they might have an aesthetic which preferred developing their values in some other direction, such as some ideal of an objective morality, wanting to have values that lead to doing most good for the world, or simply preferring independence and resolving any value conflicts purely by themselves, without any nudging from the AI. Similarly to an approval-directed agent only changing a person's brain if that was genuinely something that the person consented to, the AI here could use the person's existing values as a base for only nudging the person towards a direction that the current values endorsed being nudged towards (if any). While this paper is not focused on the question of how to combine the conflicting values of different people, it is also interesting to notice that the flexibility in resolving value conflicts may be useful in trying to find a combination of values that as many people as possible could agree on. If such a set of globally most agreeable values could be found, different AI systems could then coordinate to direct the whole global population towards that set. This might look like something akin the "Coherent Blended Volition"  (Goertzel and Pitt 2012)  proposal, which seeks to "incorporat[e] the most essential elements of [people's] divergent views into a whole that is overall compact, elegant and harmonious". In this proposal, AI systems would be used to assist people in reaching an agreeable combination of their values. Given the great degree of flexibility allowed by the possibility of social feedback loops, in which directed changes to a culture's customs could potentially cause major changes to the values of that culture, one might be very optimistic about the possibility of reaching such an outcome.
 Criteria for evaluating the framework This proposed framework is worth evaluating from two distinct, though overlapping, angles. First, is it correct? Second, to the extent that it is correct, is it actually useful for solving the value learning problem? The correctness angle suggests the following possible criteria: 1. Psychologically realistic. The proposed model should be compatible with that which we know about current human values. As a bare minimum, it should not make predictions about human behavior which would fail to correctly predict the behavior of most typical test subjects. Motivation: an agent seeking to model human values cannot be expected to get it right unless its assumptions are based on a correct model of reality.
 Compatible with individual variation. The proposed model should be flexible enough to be able to take into account the full range of variation in human psychology. It should be able to adapt it to accurately represent the values (and thus behavior) of groups as differing as Western and non-Western  (Henrich et al. 2010) , autistic and non-autistic, and so on. Motivation: psychological research often focuses on typical individuals and average tendencies, whereas a value learner should be capable of taking into account the values of everyone. 3. Testable. The proposed model should be specific enough to make clear predictions, which can then be tested. Motivation: vague models that do not make specific predictions are useless for practical design work. 4. Integrated with existing theories. The proposed definition model should, to as large an extent possible, fit together with existing knowledge from related fields such as moral psychology, evolutionary psychology, neuroscience, sociology, artificial intelligence, behavioral economics, and so on. Motivation: theoretical coherence increases the chances of the model being correct; if a related field contains knowledge about human values which does not fit together with the proposed model, that suggests that the model is missing something important. A second set of criteria is suggested by the usefulness to the value learning problem. These overlap somewhat with the "correctness" criteria in that a correct model of human value would likely also fulfill the "usefulness" criteria. However, here more emphasis is put on how well-suited the model is for answering these kinds of questions in particular: it would be possible to have a model which was capable of answering these questions in principle, but burdensome and unlikely to find the right answers in practice. 5. Suited for exhaustively modeling different values. Human values are very varied, and include very abstract ones such as a desire for autonomy and an ability to act free from external manipulation. The details of when these values would be considered fulfilled may be highly idiosyncratic and specific to the mind of the person with that value, but the proposed model should still be able of incorporating that value. Motivation: again, a value learner should be capable of taking into account the values of everyone. 6. Suited for modeling internal conflicts and higherorder desires. People may be genuinely conflicted between different values, endorsing contradictory sets of them given different situations or thought experiments, and they may struggle to behave in a way in which they would like to behave. The proposed model should be capable of modeling these conflicts, as well as the way that people resolve them. Motivation: if a human is conflicted between different values, 7. Suited for modeling changing and evolving values. Human values are constantly evolving. The proposed model should be able to incorporate this, as well as to predict how our values would change given some specific outcomes. Motivation: an AI should be capable of noticing cases where we were about to do things that our future selves might predictably regret, and warn us about this possibility.  (Yudkowsky 2004 ) A dynamic model of values also helps prevent "value lock-in", where an AI learns one set of values and then enforces that even after our values have shifted. 8. Suited for generalizing from existing values to new ones. The proposed model should be able to react to circumstances where either our understanding of the world, or the world itself, changes dramatically and forces a reconsideration of how existing values apply to the new situation. Motivation: technological and social change is constantly occurring, and often forces these kinds of reevaluations, as discussed in section 2 earlier.
 Evaluation I will now evaluate my proposed framework in light of the above criteria. 1. Psychologically realistic. The framework is motivated by psychological research, particularly in the fields of moral psychology and emotion research. However, not all of the work in the said fields has yet been fully integrated to the framework, which may bias the implications of the framework. In particular, dual-process models of morality  (Greene 2007 (Greene , 2014 , which also cover more non-emotional reasoning, are not yet a part of this framework. Dual-process models have made accurate predictions about less emotional people tending to make more "utilitarian" judgments in moral dilemmas, an outcome which the proposed framework would not have predicted. Additionally, while this framework has been developed based on psychological theories, neuroscientific evidence has not yet been considered in depth. With regard to the neuroscientific data, a specific shortcoming of the current framework is the possible existence of several different reinforcement learning mechanisms in the brain  (Dayan 2011 , Dayan & Berridge 2014 , requiring further investigation to identify the extent to which the mechanisms hypothesized here are implemented in one system or several, and how those systems interact when it comes to questions of values and morality. 2. Compatible with individual variation. In the framework, differing values are hypothesized to emerge from individual differences (some of them which are caused by cultural differences) related to things that provide positive or negative rewards. This is compatible with a great degree of individual variation. For example, various differences in moral intuitions (due either to culture or something else) can be modeled as those intuitions causing different combinations of affect in response to the same situation, with this then leading to the same concepts being associated with different levels of affect for different individuals. 3. Testable. The framework is currently insufficiently specific to make novel predictions, which will be addressed in follow-up work. 4. Integrated with existing theories. The framework is currently moderately integrated with theories of reinforcement learning, moral psychology, and emotion research. However, there remains considerable room for further integration, and there are related fields such as sociology, which have not yet been addressed at all. 5. Suited for exhaustively modeling different values. In the framework, any concept that a human may have, either on a conscious or subconscious level, can be a value. 6. Suited for modeling internal conflicts and higherorder desires. Higher-order desires can be modeled to some extent: for example, a drug addict's desire to quit a drug might be modeled as negative affect around the concept of being a drug user, or as positive affect around the concept of quitting. However, the current framework does not fully explain the existence of internal conflicts and people engaging in actions which go against their higher-order desires. For this, further theoretical integration is needed with models such as ones positing separate mental modules optimizing for either short-term or long-term reward  (Kurzban 2012) . Such integration would help explain addictive behaviors as the modules optimizing for short-term reward "overpowering" the ones optimizing for long-term reward. 7. Suited for modeling changing and evolving values. As values are conceptualized as corresponding to a value function which is constantly updated and recomputed, the framework naturally models changing values. 8. Suited for generalizing from existing values to new ones. Section 8 discussed a possible way for a value learner to use this framework for generalizing existing values into new situations, by simulating a human in different situations and modeling the amount of reward obtained from those situations, as well as using existing values to guide this simulation. To my knowledge, there have not been proposals for definitions of human values that would be relevant in the context of AI safety engineering. The one proposal that comes the closest is  Sezener (2015) . This paper takes an inverse reinforcement learning approach, modeling a human as an agent that interacts with its environment in order to maximize a sum of rewards. It then proposes a value learning design where the value learner is an agent that uses Solomonoff's universal prior in order to find the program generating the rewards, based on the human's actions. Basically, a human's values are equivalent to a human's reward function. As a comparison, an evaluation of Sezener's proposal using the same criteria follows. 1. Psychologically realistic. Sezener's framework models humans as being composed of a reward mechanism and a decision-making mechanism, where both are allowed to be arbitrarily complex, so e.g. the reward mechanism could actually incorporate several distinct mechanisms. Thus a range of models, some of them more realistic than others, could be a part of the framework. 2. Compatible with individual variation. Because both the agent and the reward function are drawn from the space of all possible programs, Sezener's proposal is compatible with a vast range of individual variation. 3. Testable. Sezener's proposal is very general, and insufficiently specific to make novel predictions. 4. Integrated with existing theories. Various existing theories could in principle used to flesh out the internals of the reward function, but currently no such integration is present. 5. Suited for exhaustively modeling different values. Because the reward function is drawn from the space of all possible programs, any value that can be represented in a computational form can in principle be represented. However, because the simplicity of the program is the only prior probability used for weighting different programs, this may not sufficiently constrain the search space towards the values that would be the most plausible on other grounds. 6. Suited for modeling internal conflicts and higherorder desires. No specific mention of this is made in the paper. The assumption of a single reward function that assigns a single reward for every possible observation seems to implicitly exclude the notion of internal conflicts, with the agent always just maximizing a total sum of rewards and being internally united in that goal. It might be possible to represent internal conflict with the right kind of agent model, but again it seems unclear whether the prior probability used sufficiently constrains the search space. 7. Suited for modeling and changing and evolving values. Because the reward function is allowed to map the same observations to different rewards at different times, the framework is in principle capable of representing changing values. However, the same problem of finding the correct function remains an issue. 8. Suited for generalizing from existing values to new ones. There does not seem to be any obvious possibility for this in the model, except if the reward function that the value learner estimates happens to generalize rewards in the same way as a human would. Overall, although both my and Sezener's frameworks represent useful progress towards a definition of human value, much work clearly remains to be done. Figure 1 . 1 Figure 1. Various source of positive or negative reward may lead to various concepts becoming imbued with reward, giving rise to both "intrinsic" values (which are valued even if the source to the original reward was severed) and instrumental values, which lose their appeal if they cease to be associated with the original source of reward.