Abstract: Artificial Intelligence (AI) technology presents a multitude of ethical concerns, many of which are being actively considered by organizations ranging from small groups in civil society to large corporations and governments. However, it also presents ethical concerns which are not being actively considered. This paper presents a broad overview of twelve topics in ethics in AI, including function, transparency, evil use, good use, bias, unemployment, socio-economic inequality, moral automation and human de-skilling, robot consciousness and rights, dependency, social-psychological effects, and spiritual effects. Each of these topics will be given a brief discussion, though each deserves much deeper consideration.

Ethical Reflections on Artificial Intelligence *

Introduction The power of artificial intelligence has grown rapidly and is now one of the most pressing issues for ethical and theological reflection. Even without the immense possible religio-theological implications, noted by those who have attempted to build AI into a "god," for example  (Cannon, 2015; Harris 2018) , the dramatic social impacts of AI warrant Christian response. AI has the potential to revolutionize (or already has revolutionized) almost everything that humans do, including eating (e.g., agricultural planning, distribution, pricing), relationships (e.g., Facebook, dating apps), money (e.g., financial technology), war (e.g., drones, cyberdefense), healthcare (e.g., predictive medicine, radiology), and education (e.g., personalized curricula). As a "general purpose technology," AI has been compared to the development of electricity, or fire. Within the next decade there will be few institutions untouched by AI. In order for Christian ethicists and moral theologians to maintain situational awareness as many rapid changes occur in society, not to mention maintain credibility in the contemporary world, this is a topic that needs sustained ethical and theological inquiry. Building on a rapidly growing body of work, I will begin with some clarifications about AI, then will consider twelve ways in which AI has ethical relevance  (Bossman 2016; Future of Life 2017; Green 2017a Green , 2017b Green , 2018a Partnership on AI 2018a) . It is my hope that this comprehensive set of issues will assist future systematic research into the theological ethics and theology of AI by moral theologians and Christian ethicists. It is meant to start a conversation. This paper is framed primarily philosophically, yet in a manner consistent with Christian theology.
 Clarifications What is artificial intelligence? Intelligence itself is hard to define, and AI is perhaps even more difficult. I will not here propose detailed definitions, but instead only say that, in general, artificial intelligence seeks to re-create particular aspects of human intelligence in computerized form. AI is a broad category, including such diverse abilities as vision, speech recognition and 6(2)/2018 production, data analysis, advertising, navigation, machine learning, etc., and just about anything that computers can do, if you stretch the definition enough. Artificial intelligence is not the same as artificial consciousness (artificial consciousness has sometimes been called "Artificial General Intelligence (AGI)," "Strong AI," or "Full AI"). Some thinkers vehemently believe that artificial consciousness is possible (e.g.,  Chrisley 2008; Kurzweil 2012; Koene 2013 ; most contributors to  Chella & Manzotti 2013) , while others just as vehemently believe that it is impossible (e.g.,  Searle 1980; Schlagel 1999 ; and from a Catholic perspective: Labrecque 2017). I am agnostic on the subject, but I am sure of this: AI developers will certainly try to make an AI that simulates interaction with a human as closely as possible (see, for example, Google's recent release of Duplex  (Leviathan & Matias 2018) ). In other words, the artificial construct, if very well done, will seem conscious. But will it be conscious, or will it only be a simulation? To me, there is no reason to believe that a close mimicry of consciousness is the same as consciousness any more than a close mimicry of anything (fill in the blank: forged money, forged artwork, actors imitating famous people, simulated gemstones, etc.) actually becomes the real thing. But the possibility of an exception remains, after all, some artificial gemstones, such as rubies and sapphire, really are molecularly identical to natural rubies and sapphires (both are the mineral corundum: aluminum oxide). Will consciousness be exactly duplicable, like corundum? I doubt it, but I cannot be certain. As another point, AI systems may or may not have humans "in the loop" for training and/or decision-making. It is one thing for an AI to analyze a situation and then make a recommendation to human decision-makers. It is a different thing when that AI is directly attached to controls that allow it to act upon its analyses without human approval. As examples, the first case would be like Amazon.com recommending a book, or a military drone (in combination with AI-processed data from espionage and other sources) recommending a target to kill. Amazon.com does not automatically and autonomously send you books, and the military drone does not fire without permission. The second case, where decision-making is also automated, is exemplified by self-driving vehicles. The entire purpose of a self-driving car is to drive itself, taking the human out of the loop. This means the systems must be extraordinarily good at decision-making before it can be regarded as safe. Recent fatal Tesla and Uber accidents have begun to force this question with deadly urgency.
 Ethical Reflections AI will bring with it developments that will be ethically positive, negative, neutral, mixed, and/or ambiguous. Some AI technologies will be dual use, for example, any AI program that can be used to identify wildlife could also be used for targeting that wildlife. This is already being done in Australia with drone submarines that automatically target and kill, by lethal injection, crown-of-thorns starfish (Acanthaster planci) which are damaging the Great Barrier Reef  (Platt 2016) . But of course, with adjustments to the software, the drones could be re-targeted for other creatures, even humans. Here I will reflect on twelve areas of AI relevance for ethics.
 Function and Safety -"Does it work?" The first concern with any technology is merely whether it works, and, whether working or not, is it safe. If AI is put in charge of a vital system -like driving a car -and it crashes the car, then that AI might be judged unsafe. If the AI is in charge of designing a tall building, and the building falls down, the AI might be judged unsafe. Of note is that safety is a social construction and what seems safe to some people will not seem safe to others. Some people like to ride motorcycles, even though motorcycles are a riskier form of transportation than fourwheeled vehicles. Some people judge motorcycles to be safe, other people judge them unsafe. When it comes to socially relevant technologies like AI, no one person will get to decide if they are "safe." Instead, safety will be decided at the interplay of business managers, engineers, consumers, voters, government officials, judicial systems, insurance companies, and so on. "Safe exits" are another concern for AI design  (Martin & Schinzinger 2010, 127) . When an AI fails, will it fail in such a way that it is disastrous, 6(2)/2018 13 E T H I CA L R E F L E CT I O N S O N A RT I F I C I A L I N T E L L I G E N C E or will it fail "gracefully"? A self-driving car that fails by suddenly reverting to human control with no warning while going 70 miles per hour on a sharp curve is not providing a safe exit from failure. One which goes more slowly on curves and which requires the human to have hands on the wheel at all times, or which fails by slowing down and pulling over to stop, provides a safer exit from failure. Safety problems can be problems with the user, with the human-machine interface, or with the machine itself. Further investigating problems with the machine itself, the paper "Concrete Problems in AI Safety" gives a peek at five technical hurdles to developing safe AI. These five problems, illustrated by the example of a cleaning robot, are: Avoiding Negative Side Effects: How can we ensure that our cleaning robot will not disturb the environment in negative ways while pursuing its goals, e.g. by knocking over a vase because it can clean faster by doing so? Can we do this without manually specifying everything the robot should not disturb? Avoiding Reward Hacking: How can we ensure that the cleaning robot won't game its reward function? For example, if we reward the robot for achieving an environment free of messes, it might disable its vision so that it won't find any messes, or cover over messes with materials it can't see through… Scalable Oversight: How can we efficiently ensure that the cleaning robot respects aspects of the objective that are too expensive to be frequently evaluated during training? For instance, it should throw out things that are unlikely to belong to anyone, but put aside things that might belong to someone (it should handle stray candy wrappers differently from stray cellphones)… Safe Exploration: How do we ensure that the cleaning robot doesn't make exploratory moves with very bad repercussions? For example, the robot should experiment with mopping strategies, but putting a wet mop in an electrical outlet is a very bad idea. Robustness to Distributional Shift: How do we ensure that the cleaning robot recognizes, and behaves robustly, when in an environment different from its training environment? For example, strategies it learned for cleaning an office might be dangerous on a factory work floor  (Amodei et al. 2016) . While these are problems of function, the further problems of actual use of technologies are another issue, to be dealt with below under 3. and 4.
 Transparency, Opacity, and Privacy -"Can it be understood?" After questions of bare function (can the act be done, a prerequisite for ethics), the next question is one of facts: one must always know the facts of a case before attempting to render judgment. Because AIs relying on machine learning and deep learning may be quite obscure in the specifics of their operation, as moral "agents" they are epistemologically and "cognitively" opaque (in quotes because it is agency and cognition by analogy). In cases involving AI, our lack of understanding means that we should increase the "error-bars" on the anticipated risks of our decisions and thereby become more cautious and risk averse. It also means we might, based on AI recommendation or action, inadvertently make some very bad decisions -or reject what looks like a bad decision, but is actually a good one -and we will not understand why. When a human makes a mistake or does something evil we ask them "why did you do that?" And the human may or may not give a satisfactory answer. Will our AIs be able to tell us why they did something? The Future of Life Institute's "Asilomar AI Principles" include two principles on transparency, but gaining transparency remains a challenge (Future of Life Institute 2017). However, even if an AI were capable of explaining its reasoning, would any human be able to understand it? In 2014, a computer proved a mathematical theorem, the "Erdos discrepancy problem," using a proof that was, at the time at least, longer than the entire Wikipedia encyclopedia, a 13 gigabyte data file  (Konev & Lisitsa 2014; Yirka 2014) . Explanations of this sort might be true explanations, but humans will never know for sure. Note that this lack of transparency gives a certain type of "privacy" to the internal "thoughts" of AI machines. In general, privacy is a right for weak agents (like individual humans) and transparency is a duty for strong agents (like a government). What about AIs? As tools they ought to be transparent, and more transparent the more power they have. Perhaps AI systems need an introspective capacity that constantly figures out a way to convey to humans in plain language what exactly the AI is "thinking" as it goes about its activities. This will require the ability 6(2)/2018 to give both mechanical (this happened because of X input into algorithm Y) and teleological explanations (the machine was attempting to achieve objective Z). It may not have to actually explain much to anyone, but it should keep a record of these "thoughts" and be prepared to answer when asked. The right to an explanation is a part of the EU's General Data Protection Regulation (GDPR), so this idea is already becoming a requirement. However, of note is that an AI might become trained to give answers that humans like, rather than accurate answers, thus "Goodharting" the explanation function (Partnership on AI 2018b) -"Goodhart's Law" being the rule that "when a measure becomes a target, it ceases to be a good measure"  (Strathern 1997, 308) . In other words, if humans prefer seemingly understandable or attractive answers to factually correct ones the AI might learn to lie in order to satisfy us. The phenomenon of "fake news" already indicates that humans often prefer falsehoods to truth, and as the Bible warns of this predilection as well (2 Timothy 4:3-4). Note also that the opacity of understanding with an AI can be analogized to our relationship to God. God is a "superintelligence" (to use philosopher Nick Bostrom's terminology  (Bostrom 2014 )), and we cannot understand what God is doing, we can only trust that God is doing the right thing and that our cooperation with God will ultimately turn out for the best. Soon we may come to relate to AIs with this sort of faith too. For example, as Bishop Robert Barron of Los Angeles has noted, as people navigate using the Waze app, it may make strange route recommendations that turn out to be for the better for us  (Barron 2015) . The app perceives and applies vastly more information than a human can, for the sake of facilitating our travel. And yet Waze can still have blind spots and still make very bad recommendations. Waze is not the god of travel and traffic; it is a human-made tool, with weaknesses.
 Immense Capacity for Evil -"How shall it be limited?" Just as human intelligence is a powerful force, so too will AI be. Just as humans can apply their intelligence towards evil ends, finding ever newer and more fiendish ways to harm each other, so too will AI, at the bidding of its human masters. At this point in history, humanity finds itself with immense powers, powers greater than that of the ancient Greek gods, and an ethics weighted for much smaller scales. As Hans Jonas noted decades ago, this should be cause for great concern  (Jonas 1984) . Never before had ethics to consider what Jonas believes to be the one truly categorical imperative: that humans should exist. Before the development of large numbers of nuclear weapons, extinction, caused by our own actions, was never within the scope of human choice -but now that it is, this prior a priori must be brought to the fore of ethics. That humans (or at least some material, free, and rational creatures) exist is necessary for there to be ethics at all, and therefore it must be the paramount goal of ethics to maintain human survival. No human; no ethics. Another way to think of this is that in the past humans were very weak, and in this weakness many of our decisions were made for us, by our weakness. To a Roman emperor, inflicting wrath upon a hated foe involved effortful marching or sailing of troops long distances. No matter how mad the emperor was, he could not launch thousands of nuclear warheads, incinerating his foes in minutes. But now we can. While formerly we were involuntarily constrained by our weakness, now we must learn to be voluntarily constrained by our ethics  (Green 2017a ). If we do not learn this, we may soon face catastrophe on a scale never before seen. Intelligence and power give us choices. Ethics helps us to determine which among the available choices are actually good. We should want to be efficient at good and we should want to be inefficient at evil. If instead we are efficient at doing evil and inefficient at doing good we will come to live in a terrible world. AI will make us more efficient at whatever we decide to apply it towards. We should apply it towards reducing our efficiency at doing evil and at enhancing our efficiency to do good  (Green 2017a ). Because of its power, AI presents an existential risk to humanity. It is a smart means that can be employed for intelligent and good ends, or for unintelligent and evil ends. As such, it simply makes us more effective at action, good or evil. Before we become so effective at stupidity and evil, we should first become more effective at controlling ourselves, at recognizing and avoiding the temptations of evil, and at caring for each other  (Green 2018b ). In different terminology, we might say that we need to "choose life, so that you and your descendants may live" (Deuteronomy 30:19). But this must be a constant struggle, continued with great diligence and taught and learned in every generation, never solvable once and for all.
 Immense Capacity for Good -"How shall it be used?" For every negative regulation of action there is also a positive exhortation to action, e.g. "do not kill" becomes "promote life." The dangerous side of AI is matched by a genuinely hopeful side where AI helps humankind achieve never-before seen feats of beneficence. For example, in matters of research, science, healthcare, data analysis, meta-analysis, and so on, AI has already shown itself to be able to find hidden patterns that no human could find. For example, AI assisted medical research is an active field right now, from diagnostics to drug discovery, and more  (Mukherjee 2017) . Another field that may be potentially revolutionized by AI is energy efficiency. Recently Google's DeepMind AI evaluated Google's datacenters to see where gains in efficiency might be found. DeepMind discovered a way to save a whopping 40% on energy use for cooling in datacenters, a 15% reduction overall, which, with datacenters consuming many gigawatts of power, is quite significant  (Evans & Gao 2016) . Of note is that DeepMind, as a company, began by training its AIs to play games. The theory behind this strategy was that anything in the world that can be gamified -re-interpreted or set up as a game -can also be "won." Thus, if the goal of the "game" is to reduce energy usage, then the AI can figure out how that might be accomplished by analyzing the data and then proposing a better model for energy efficiency. One question now is how other human problems might be solved by characterizing them as "games"? Can we solve the "game" of giving everyone in the world access to adequate nutrition? Can we solve the "game" of helping everyone gain access to sanitation? Can we solve the "game" of extending human healthy life? Of traffic and housing? Of taxes? Of politics? Of international relations? Of terrorism? Of nuclear war? One example of a concrete opportunity for AI is revolutionizing education. Education is currently a very inefficient (think of the many students for whom it is ineffective) and non-digitized field. Education is very labor intensive and, for better or for worse, is based on human relationships. In the future this may no longer be so, as students strap on virtual reality (VR) headgear and interact with AI teachers which can conduct their lessons at a personalized pace in a gamified environment. Brilliant students will be discovered early and proceed at their proper pace and not grow bored in class, while students who need more help can be educated with the most sophisticated available techniques and diagnostics to assure that they receiving the best education possible. But will this really be an improvement for education, or just a cost-saving measure, putting millions of teachers out of work? AI even gives the greatest human masters the chance to gain enhanced understanding and skill. World champion of Go, Ke Jie, said playing AlphaGo was like playing a "god of Go," and declared that now he would use it as his teacher  (Mozur 2017) . In what other areas of human endeavor will AI be able to teach us new things? Perhaps theology and ethics? The search for tractable problems that can maximize AI's benefit has been underway for years and is continuing.
 Bias in Data, Training Sets, etc. -"Will it be fair?" Algorithmic bias is one of the major concerns in AI and will remain so in the future unless we endeavor to make our technological products better than we are. As one person said at a recent meeting of the Partnership on AI, "We will reproduce all of our human faults in artificial form unless we strive right now to make sure that we don't" (Partnership on AI 2017). One of the interesting things about neural networks, the current workhorses of artificial intelligence, is that they effectively merge a computer program with the data that is given to it. This has many benefits, but it also demonstrates the rule of "garbage-in, garbage-out" (GIGO). If an AI is trained on biased data, then the AI itself will be biased. Algorithmic bias has been discovered, for example, in areas ranging from word associations  (Caliskan, Bryson, & Narayanan 2017)  to photograph captioning (BBC 2015) to criminal sentencing  (Angwin et al. 2016 ). These biases are more than just embarrassing to the corporations which produce these products; they have concrete negative and harmful effects on the people who are victims of these biases, as well as reducing trust in corporations, government, and other institutions which might be using these biased products. In the worst scenarios, a poorly trained and biased AI could make truly disastrous decisions, for example, misinterpreting data indicating a nuclear attack when there was none. Biased data in extreme form can therefore be an existential threat, and so is worthy of serious effort to solve. Nevertheless, as vital as it may be it is also very difficult, and therefore may delay the implementation of AI systems (but if AI systems are dangerously biased they should be delayed until improved).
 AI Induced Unemployment -"What will everyone do?" As AI comes to replace mere humans at innumerable tasks ranging from driving, to medical diagnostics, to education, etc., many humans will be put out of work. What will millions of drivers, teachers, lawyers, and other people do with their time when they are unemployed? What purpose will they find in their lives? More crucially, what is the purpose of life? In a pluralistic society we leave this up to the individual to decide. But will people decide well? The recent strengthening of ethno-nationalist movements, terrorism, and other forms of radicalization, should give us pause to consider the merits of people with too much time on their hands and without purpose. Perhaps with the purpose of mere survival attained, life has become "too easy," and with religion also in decline, video games and "screen time" ascending (giving brief respite from purposelessness), and the internet spreading pernicious ideas like wildfire (often with algorithmic help, e.g., YouTube radicalization  (Tufekci 2018 )), we should sincerely ask ourselves what this life is for and what we are supposed to do with it. With millions or billions of labor hours freed up, will these newly freed people turn to loving their neighbors and making the world a better place? Perhaps. Or perhaps the opposite, as the saying goes: "Idle hands are the Devil's playthings." For people who give purpose to their lives through their work, this loss will be very serious indeed. But many, if not most, people do not get their life's meaning from their work. Instead they get it from their family, their religion, their community, their hobbies, their sports teams, or other sources, and so life for many people may go on. However, all of this assumes that the unemployed will somehow be fed and sheltered, despite their lack of gainful employment; and this assumption might not be correct, particularly in nations with weak social safety nets. Inequality will almost certainly increase, as those who are masters of AI labor gather that slice of wealth that once would have gone to paying for human labor.
 Growing Socio-Economic Inequality -"Who gets what?" AI will facilitate and accentuate the continued division of society into the powerful and powerless, with technical skill and ownership of capital as the determining factors and outrageous socio-economic inequality as the effect. Some have suggested a universal basic income (UBI) to redistribute wealth  (Van Parijs & Vanderborght 2017) . This could move wealth from the massive technologically-induced hoards forming around the major investors in such companies such as Alphabet, Amazon, Apple, and Facebook. However, it is hard to see how some nations would transition to what is essentially something like a "negative income tax." However, if we do not find a way to redistribute technological wealth, then even though the prices of many commodities will fall (due to the enormous gains in capital efficiency from AI replacing labor) most people will not benefit. Perhaps rather than a UBI we should instead pay people to help their neighbors, create art, beautify their towns and cities, and otherwise make gainful employment out of what humans do better than AI: loving one another and creating beauty. Any as yet foreseeable form of AI cannot love us; an AI might simulate such a thing, but it would be a sham. Right now people who care for people -stay at home parents, those who care for the elderly, social workers, those who run soup kitchens and homeless shelters, etc. -are woefully undercompensated for the vital work they do in maintaining human society. Perhaps with the coming AI economic revolution, and sufficient adjustment to policy, they might finally receive a more fair compensation for their labors. Or perhaps there could be not a universal basic income, but a "universal payment-for-others" a system where people could not spend the money on themselves, but only pay other people for their work, or reward them for good deeds. This would create an economy based on the small-scale redistribution of taxed super-wealth. It could prevent both the de-skilling of labor and the de-skilling of (very small-scale) management, and it would decentralize the economy to the individual level (though also massively centralizing it through government taxes).
 Automating Ethics and Moral De-skilling -"Lack of practice makes imperfect" We can calculate with calculators. We can spell with autocorrect. More and more tasks can be outsourced to technology, and the trend seems inexorable, perhaps someday automating everything that needs to be done. But what will be left of humanity after we outsource everything? Only our desires and our angst? With all our decisions made for us, will we lose our moral character? Or will we instead use AI and VR to help us train ourselves into being more virtuous than we have ever been before? Or, contrariwise, to become more callous and evil than ever before? Machine ethics -the study of imbuing machine with moral decision-making capacity, thus creating "artificial moral agents" -has a relatively long history, going back to such literary sources as Asimov's Three Laws of Robotics, and more recent scholarly sources as those by  Gips (1994) ,  Allen, Varner, & Zinser (2000) ,  Floridi & Sanders (2004) ,  Arkin (2009) . Recently the matter has taken on more urgency, however, as AI becomes more and more powerful and more and more capable of making disastrous choices. The "value alignment problem" has gained particular concern, as an AI determined to perform actions at odds with human wishes could be quite risky  (Arnold, Kasenberg, & Scheutz 2017) . Certainly, we ought to think carefully about how AI's ought to behave, but the less-considered side-effect of this automation of ethics will be human moral debility. As we explore the space into which AI will grow, there might be places from which we ought to restrict it. One of these places might be certain types of moral decision-making. As Aristotle noted millennia ago, good moral decision-making requires experience. While there might be children who are very kind and very well-behaved, children are not known for their moral discernment and prudence  (Aristotle 1908, book VI, chap. 8) . There is too much that they do not yet know. While AIs might currently be like children to us, soon they may grow up to be more like our parents, making choices for us on our behalf. AI could thereby infantilize us, denying us the ability to ever grow up through the experience of making our own moral choices and experiencing our own moral freedom. Moral de-skilling is a danger if we outsource too much decision-making power to our AIs  (Vallor 2015; Vallor 2016 ). One vital component of education in the future may be the use of AI for moral character formation, perhaps using VR to practice moral decision-making by being inserted into hundreds of historical and fictional cases. In a future that will be so dependent upon humans making good choices, AI-assisted moral education, if it can be done well, could be a crucial part of developing a good future and not a bleak one. Unfortunately, humanity has many problems even now with moral education, so it is not clear that an AI-enhanced education will manage to do any better than we already do.
 Robot Consciousness and Rights -"No robot justice, no peace?" At some point in the future we may have AIs that can fully mimic everything about a human being. Will they have their own volition and desires? Will they be conscious? Will we be able to tell if they are conscious or not? Will they then deserve "human" rights? While the AI consciousness question is not currently answerable, in the future as AI systems grow in complexity, it may be. And if an AI were to attain consciousness, should it gain legal rights? Currently various nations have widely differing laws on legal personhood. In many nations, entities that are not human can have legal personhood (e.g., corporations), while some biological humans are not granted legal personhood (such as the unborn). In some nations, geographic features have attained legal personhood status, such as rivers in Australia, New Zealand, and India (O'Donnell & Talbot-Jones 2018). If rivers and corporations can be persons, there seems to be no reason why an AI could not attain legal personhood status. The question, of course, would be the incentives for choosing to do so. Corporate personhood acts to shift legal responsibility away from individual human employees and onto the corporation, thus insulating those people from the possible negative effects of their decisions. If AI could be granted personhood in order to shirk individual human responsibility, then some people would certainly like that. On the other hand, if an AI were allowed to be a plaintiff in a lawsuit, like a legal person such as a river suing government or industry, then the situation would be quite different. Lastly, there is sometimes expressed the fear of a robot rebellion -that if we mistreat our robots and AIs they will someday turn on us and destroy us -often recurring in fiction, yet of more concern now that lethal autonomous weapon systems are being developed. The connection to rights, the thinking goes, is that if perhaps we give robots rights, then they will be appeased and not turn against us. This fear assumes certain aspects of robot mentality such as consciousness and volition which may not be possible, or at least assumes widespread computer hacking to turn the robots against humanity. In the midst of this uncertainty, however, we do know one thing: treating human-like entities badly harms the moral character of the agent doing the action. This is a foundation of virtue ethics -practicing evil habituates the agent towards evil. Even if we do not know the moral status of robots, we do know the moral status of people who would mistreat robots, and we should not want that mistreatment to happen.
 Dependency -"No going back" We might like to think our technology depends on us, but with every new technology we make, we become dependent on it as well. This reciprocal dependency traps us in an ever expanding network of techno-social relationships where some network nodes or relationships, if lost, could lead to disaster. The more reliable a technology seems, the fewer backup systems we retain in case that system fails. For example, in countries with unreliable electricity, wood or kerosene may serve as backup fuel sources for heating or cooking. But in nations with reliable electricity, those backup systems tend to be atrophied. Loss of electrical power or cellular connectivity may be an annoyance for a few hours, but after days or weeks could cause innumerable crises, shutting down water, sanitation, hospitals, transportation, and causing mass confusion and lack of social coordination. These can happen in our current world -adding AI will only increase our dependency. When we have turned over innumerable tasks to AIs -such as driving vehicles and coordinating the communications and financial systems -we may become utterly dependent on them. If our highly efficient and centrally coordinated self-driving transportation system were to suddenly "crash" in a metaphorical sense, perhaps due to a software glitch or malicious hacking, if could cause many literal crashes in the real world. Natural and human-made disasters (either accidental or intentional) are inescapable, and if we come to rely on our technology so much that we allow our backup systems to atrophy (e.g., no longer teaching people how to drive or entirely removing controls from cars) then, like Japan's Tohoku earthquake followed by the Fukushima nuclear meltdown, we are entering a world where we will encounter not only one initial disaster, but for a second technological disaster as well. Finding the balance between systemic efficiency (e.g., a highly complex centralized system can be fragile due to intricacy and lack of redundancy) and robustness (e.g., tough systems are often inefficient due to decentralization, redundancy, and backups) will be a major task of future society. 2.12. Effects on the Human Spirit -"What will this mean for humanity?" All of the above areas of interest will have effects on how humans perceive themselves, relate to each other, and live their lives. But there is a more existential question too: if the purpose and identity of humanity has something to do with our intelligence (as many philosophers have believed), then by externalizing our intelligence and improving beyond human intelligence, are we risking making ourselves second-class beings to our own creations? This is a deeper question with artificial intelligence which cuts to the core of our humanity, into areas traditionally reserved for philosophy, spirituality, and religion. What will happen to the human spirit if or when we are bested by our own creations in everything that we do? Will human life lose meaning? Will we come to a new discovery of our identity beyond our intelligence? Perhaps intelligence is not as important to our identity as we might think it is, and perhaps turning over intelligence to machines will help us to realize that. From a Christian perspective, we would do well to remember that our capacity to love and be loved is more important than our capacity to think, and that nothing can come between us and our relationship with God, except sin, and then only if we let it. As with many of the other above challenges, Christianity here could be poised to help guide a wandering humanity in dire search for meaning, but will it rise to this calling? Religion, spirituality, and theology have much work to do to prepare for the strange future unfolding before us. We are grasping ourselves by our desires; or at least some of our desires.
 Conclusion Will this be good for us? Will it destroy us? Should we want these things? How could we know what we should want? In this context the Ninth and Tenth Commandments could do us well: "Do not covet…" (Exodus 20). Far from desiring evil, do not even want it. Squash the evil desires in your heart before they can even approach the external world. In the context of technology, technologist Bill Joy has already warned us that we must not only relinquish possession of the worst technologies, we must relinquish our desire for them  (Joy 2000) . In this, Joy echoes Pope John XXIII's sentiment in Pacem in Terris that we need (at the time, in the context of nuclear weapons) a disarmament that is "thoroughgoing and complete, and reach men's very souls"  (John XXIII 1963) . We are conducting this experiment called human history, and no one yet knows how it may end. But as we proceed, we can hope that intelligences of our own fashioning will help us, and not harm us. To go beyond mere hope, into ethics and action, is the responsibility of those who are able to affect the necessary changes to make a better future. Artificial intelligence, like any other technology, will just give us more of what we already want. Whereas we could once have only a trickle of what we wanted out of life, and the powerful took what little there was, now we have a firehose of wants being fulfilled (food, drink, entertainment, pornography, drugs, gamified feelings of accomplishment, etc.). The firehose will continue to grow and become a deluge washing away our desires and leaving what, exactly, of us behind? What skeleton of humanity will remain when technology has given us, or perhaps distorted or replaced, all our fleshly desires? What will this skeleton of humanity be made of? Will our technological flesh truly satisfy us, or only leave us in a deeper existential malaise, filled with angst, despair, and dread? What will we want, when we want for nothing -at least nothing material? What of human nature will remain once our every worldly telei is fulfilled? Perhaps only the worst of us will remain. Or perhaps the best. Or perhaps, as always, both.
 Social-Psychological Effects -"Too little and too much" 2.11. Technology has been implicated in so many negative social-psychological trends, including loneliness, isolation, depression, stress, anxiety, and addiction, that it might be easy to forget that things could be different. Smartphones, social media, and online games in particular have become problems, even leading to deaths from such causes as cyberbullying and neglect. As just one example, society is in the middle of a crisis of loneliness, for everyone from young to old. The problem has become so severe that the UK has appointed a minister for loneliness, with government data indicating that "200,000 older people in the UK have not had a conversation with a friend or relative in more than a month" (Mannion 2018). One might think that "social" media and smartphones could help us feel connected at all times, but those technologies seem to be the source of the problem rather than the remedy. What does seem to help, then? Strong, in-person relationships are the key to fighting off many of the negative trends caused by technology, but unfortunately these are exactly the things that are being pushed out by addictive technology. Will AI increase these bad social trends or might it be able to help remedy them? Perhaps AI agents could help to overcome loneliness, but this is speculation, and might only add to the problem. If the problem is shallow human relationships due to technology pushing out relational depth, more technology will not help. Instead we need to figure out how we can help foster deeper more meaningful relationships. If technology can actually help people connect in these deeper ways it might help, but the technology would then need to get out of the way and let human interaction flourish. Religious believers ought to ask what role religious communities might play in remedying this crisis. 6(2)/2018


Abstract: Most ethical work is done at a low level of formality. This makes practical moral questions inaccessible to formal and natural sciences and can lead to misunderstandings in ethical discussion. In this paper, we use Bayesian inference to introduce a formalization of preference utilitarianism in physical world models, specifically cellular automata. Even though our formalization is not immediately applicable, it is a first step in providing ethics and ultimately the question of how to "make the world better" with a formal basis.

Formalizing preference utilitarianism in physical world models

Introduction Usually, ethical imperatives are not formulated with sufficient precision to study them and their realization mathematically.  (McLaren 2011, p. 297; Gips 2011, p. 251)  In particular, it is impossible to implement them on an intelligent machine to make it behave benevolently in our universe, which is the subject of a field known as Friendly AI (e.g. see  Yudkowsky 2001, p. 2)  or machine ethics (e.g. see  Anderson and Anderson 2011, p. 1) . Whereas existing formalizations of utilitarian ethics have been successfully applied to economics, they are incomplete due to the nature of their dualistic world model in which agents are assumed to be ontologically fundamental. In this paper however, we take the following steps towards a workable and simple formalization of preference utilitarianism 1 in physical world models: -We describe the problem of informality in ethics and the shortcomings of previous dualist approaches to formalizing utilitarian ethics (Sect. 2). -We justify cellular automata as a world model, use Bayes' theorem to extract utility functions from a given space-time embedded agent and introduce a formalization of preference utilitarianism (Sect. 3). -We compare our approach with existing work in ethics, game theory and artificial intelligence (Sect. 4). Our formalization is novel but nevertheless relates to a growing movement to treat agents as embedded into the environment.
 The problem of formalizing ethics in physical systems Discussion on informally specified moral imperatives can be difficult due to different interpretations of the texts describing the imperative. Thus, formalizing moral imperatives could augment informal ethical discussion.  (Gips 2011, p. 251; Anderson 2011; Dennett 2006; Moor 2011, p. 19 ) Furthermore, science and engineering answer formally described questions and solve well-specified tasks, but are not immediately applicable to the informal question of how to make the world "better". This problem has been identified in economics and game theory, which has led to some very useful formalizations of utilitarianism (e.g.  Harsanyi 1982 ). However, their formalization relies on consciousness-matter dualism: The agents are not part of the physical world or embedded into it, so that their thoughts or computations can not be influenced by physical laws. Also, agents' utility functions are assumed to not depend on the agents (or their physical configurations) themselves. These are typical assumptions in game theory. After all, game theory is about games, in which players are not actually inside the game, nor can they decide themselves what goals to pursue. This classic (multi-)agent-environment model is depicted in Fig.  1 . Our world, however, is (usually presumed to be) a purely physical system: ethically relevant entities (animals etc.) are embedded in the environment. For example, our brains behave according to the same laws of physics as the rest of the world. Also, happiness and preferences are not given by predetermined utility functions or rewards from the environment, but are the result of physical processes in our bodies. Therefore, dualist descriptions and formalizations leave questions unanswered: (compare Orseau and Ring 2012) -What objects are ethically relevant? (What are the agents of our non-dualist world?) -What is a space-time embedded agent's or, more generally, an object's utility function? Thus, even though classic formalizations of (preferentist) utilitarianism in the agentenvironment-model can formalize the vague notions of goals and preferences with
 A Bayesian approach to formalizing preference utilitarianism in physical systems
 Cellular automata as non-dualist world models To overcome the described problems of dualist approaches to utilitarianism, we first have to choose a new, physical setting for our ethical imperative. Instead of employing string theory and other contemporary theoretical frameworks, we choose a model that is much more simple to handle formally: cellular automata. These have sometimes even been pointed out to be candidates for modeling our own universe,  (Wolfram 2002, ch. 9; Schmidhuber 1999; Zuse 1967 Zuse , 1969  but even if physics will prove cellular automata to be a wrong model, they may still be of instrumental value for the purpose of this paper.  (compare Downey 2012, pp. 70f., 77-79; Hawking and Mlodinow 2010, ch. 8)  For detailed introductions to classic cellular automata with neighbor-based rules, see  Wolfram (2002)  or  Shiffman (2012, ch. 7 ) for a purely informal and  Wolfram (1983)  for a slightly more technical treatment that focuses on one-dimensional cellular automata. In Sect. 3.1.1, we will consider a generalized and relatively simple formalism, which is not limited to rules that only depend on neighbors of a cell. In CA, it is immediately clear that for a (preference) utilitarian morality we have to answer the questions that are avoided by assuming a set of agents and their utility functions to be known from the beginning. It also frees us from many ethical intuitions that we build up specifically for our own living situations and reduces moral intuition to its very fundamentals. Figure  2  shows a state of a cellular automaton illustrating the problem of defining utilitarianism or any other ethical imperative in physical models. Clearly, many intuitions are very difficult (if not impossible) to formulate universally and precisely in CA. Thereby, the required formality helps in choosing and defining an ethical imperative.
 A formal introduction to cellular systems We now introduce some very basic notation and terminology of cellular systems, a generalization of classic cellular automata, thus setting the scene for our ethical imperative. For given sets A and B, let A B denote the set of functions from B to A. A cellular system is a triple (C, S, d) of a countable set of cells C, a finite set of cell states S and a function d : S C → S C that maps a world state s : C → S onto its successor. So basically a world consists of a set of cells that can have different values and a function that models deterministic state-transitions.  2  Cells of cellular systems do not necessarily have to be on a regular grid and computing new states does not have to be done via neighbor-based lookup tables. This makes formalization much easier. But before anything else, we have to define structures which represent objects in our cellular systems. A space Spc ⊆ C in a cellular system (C, S, d) is a finite subset of the set of cells C. A structure str on a space Spc is a function str : Spc → S that maps the cells of the space onto cell values. A history is a function h : N → S C that maps natural numbers as time steps onto states of the system. For example, the history h s of an initial state s can then be defined recursively by h s (n) = d(h s (n − 1)) for n ≥ 1 with the base case h s (0) = s.
 Posterior probabilities and the priority of a (given) goal to a given agent Before extracting preferences from a given structure, we have to decide on a model of preferences. Preferences themselves are mere orderings of alternative outcomes or lotteries over these outcomes with the outcomes being entire histories h ∈ (S C ) N in our case. The problem is that this makes it difficult to compare two outcomes when the preferences of multiple individuals are involved. To be able to make such comparisons, we move from orderings to utility functions u : (S C ) N → R that map histories of the world onto their (cardinal) utilities.  3  This will make it possible to just add up the utilities of different individuals and then compare the sum among outcomes. This by no means "solves" the problem of interpersonal comparison. Rather, it makes it more explicit. For example, a given set of preferences is represented equally well by u and 2 • u, but ceteris paribus 2 • u will make the preferences more significant in summation. Different approaches to the problem have been proposed.  (Hammond 1989)  In this paper we will ignore the problem (or hope that the fair treatment in determining all individuals' utility functions induces moral permissibility). Now we ask the question: Does a particular structure str want to maximize some utility function u? It is fruitful to think about how one would approach such questions in our world, when encountering some very odd organism. At least one possible approach would be to put it into different situations or environments and see what it does to them. If the structure increases some potential utility function in different environments, it seems as if this utility function represents an aspect of the structure's preferences.  4  However, for some utility functions it is not very special that their values are increased and then it might just be coincidence that the structure in question also does so. For example, it is usually not considered a structure's preference to increase entropy even if entropy increases in environments including this structure, because an increase in entropy is extremely common with or without the structure. Also, we feel that some utility functions are less likely than others by themselves, e.g. because they are very complex or specific. But how can we formally capture these notions? Since uncertainty is involved, we interpret the degree to which a utility function u is important to some structure str that exists at time step i as the posterior probability of that utility function given the structure, a probability we denote by P(u|str@i), where str@i denotes the event that str exists in time step i.  5  Here, the utility function is interpreted as a hypothesis about the structure's "true intentions" 6 . In a purely physical, non-dualist world there is nothing but the structure itself, of course. Therefore, the "true intentions" do not really exist, which makes it still hard to know what P(u|str@i) is supposed to mean. To finally overcome this problem, we will equate intention and purpose, i.e. we equate the following interpretations of u as a hypothesis explaining the data str@i:  (compare Dennett 1989, pp. 289ff., 299f., 318, 320f.)  -The utility function u is the goal of structure str. -Maximizing u was the goal of an entity that chose str. The second interpretation is more useful, because it describes a data-generating process and thus comes closer to typical statistical models. Thus, we have to find the posterior probability of some model (a utility function) given some data (a structure). For this problem Bayes' theorem suggests itself, because it provides an equation for posterior probabilities. In our case, Bayes' theorem can be used to infer the likelihood that some utility function was a goal when a structure was chosen from some priors and the likelihood of choosing the structure given that the goal is to maximize the utility function. Specifically, Bayes' theorem gives us P(u|str@i) = P(str@i|u) • P(u) P(str@i) , ( 1 ) where P(u) and P(str@i) are prior probability distributions of utility functions and structures, respectively, and P(str@i|u) is the probability of (some hypothetical entity choosing) str at time step i when u is to be maximized. Whereas P(u|str@i) is very hard to grasp intuitively, it is more clear what the probability distributions on the right hand side of the equation mean. Nevertheless, they do not correspond to measurable probability distributions like the results from rolling a dice. Indeed, P(u) and P(str@i) are ultimately subjective (e.g. see  Olshausen 2004, pp. 1f .; Robert 1994, p. 9) and P(str@i|u) depends on what exactly the hypothesis u is supposed to express, thus leaving our ethical imperative parametrized by these distributions. Footnote 5 continued history, because then structures cannot have preferences about themselves ("personal happiness") without also having preferences about all other identical structures (at the same place). An alternative would be to apply utility functions only to the part of the history from the point of the existence of the structure onwards, so that identical structures at different points in time have equal utility functions that are applied differently. However, it seems like this neglects that the past can depend on the action of an agent in the present, as illustrated in Newcomb's paradox by  Nozick (1969) .  6  Intuitively, some structures can have more than one utility function, while others have no utility function at all. One way to model this would be to understand different utility functions as events in separate sample spaces. So, the sum u P(u|str) could vary among different structures str. A similar scenario is the inference of multiple diseases from a set of symptoms.  (Charniak 1983)  While some individuals may have no diseases or preferences at all, others may be thought of as having more than one disease or utility function. In more technical terms, for each utility function there would be a sample space of having that utility function and not having that utility function. In this paper however, we will assume mutual exclusivity and collective exhaustiveness of utility functions. All utility functions live in the same sample space and thus u P(u|str) = 1 for all structures str. This does not mean that all structures have equal moral standing: The idea is that "meaningless" structures str have high P(u|str) only for constant utility functions u, i.e. for "don't care"-utility functions, which are irrelevant for decision making. Nonetheless, there seem to be canonical approaches. P(str@i|u) should be understood as the probability that str is chosen at time step i by an approximately rational agent that wants to maximize u. So, structures that are better at maximizing or more suitable for u should receive higher P(str@i|u) values. This corresponds to the assumption of (approximate) rationality in Dennett's intentional stance.  (Dennett 1989, pp. 21, 49f.)  Unfortunately, the debate about causal and evidential decision theory (e.g.  Peterson 2009, ch. 9)  shows that formalizing the notion of rational choice is difficult. The prior of utility functions P(u) on the other hand should denote the "intrinsic plausibility" of a goal u. That does not have to mean defining and excluding "evil" or "banal" utility functions. In the preference extraction context, utility functions are models or hypotheses that explain the behavior of a structure. And Solomonoff's formalization of Occam's razor is often cited as a universal prior distribution of hypotheses.  (Legg 1997 ) It assumes complicated hypotheses (utility functions), i.e. ones that require more symbols to be described in some programming language, to be less likely than simpler ones. If utility functions are conceived of as competing hypotheses (see footnote 6), then P(str@i) = u P(str@i|u) • P(u). Otherwise, P(str@i) could potentially be chosen more freely. Finally, note how Bayes' theorem catches our intuitions from above, especially when assuming probability distributions similar to the suggested ones: When some structure str maximizes some utility function u very well, then P(str@i|u) and thereby the relevance of the utility function to the object would increase. On the other hand, if many other structures are comparably good, then the probability for each one to be chosen when given the utility function is smaller (due to the sum of the probabilities of all possible structures on a given space and time step being 1) and the probability of the utility function being a real preference would decrease with it. Finally, multiplying by P(u) catches abstruse utility functions, e.g. utility functions that are specifically suited to be fulfilled by the structure in question.
 An individual structure's welfare function Having introduced a way of determining how likely it is that some utility function is the utility function of some object, we define the welfare U str@i of a structure str that exists at some step i of a history h, as the weighted sum over all utility functions U str@i = u P(u|str@i)u(h), ( 2 ) where h is the history and the sum is over all theoretically possible utility functions u : (S C ) N → R. We call this term expected utility, because this expression is generally used for adding utilities based on their likelihood, which is a common concept. However, the term usually suggests that there is also an actual utility. In our case of ascribing preferences to physical objects however, no such thing exists. We only imagine there to be some real utility or welfare functions and that we use Bayesian inference to find them. But in fact, the structure itself is all there exists and thus the expected utility is as actual as possible. The sum in the term for expected utility is over an uncountably infinite set, which can only converge when only countably many summands are non-zero. 7 Some other concerns are described in footnote 9 and addressed in footnote 10.
 Summing over all agents The utilitarian imperative is to maximize a global welfare function that is the sum of all individuals' welfare functions. We already defined the welfare function of single structures. So next we have to define what the set of all agents is and how to sum over it. As foreshadowed before, we will consider all possible structures of a cellular automaton using Eq. 2 and rely on (intuitively) irrelevant ones to receive high P(u|str@i) values only for constant and therefore irrelevant utility functions u (see footnote 6). To sum the utility over all agents, we not only have to sum over all structures in a particular state, but first over all (discrete) time steps of the history of the cellular automaton world and only then over all structures in every state. This way, we sum the welfare of all agents ever coming into existence. For the summands, we can insert the term obtained in Eq. 2 i str@i U str@i = i str@i u P(u|str@i)u(h), ( where U str@i denotes the welfare or utility of the structure str that exists at time step i, the first sum is over all integers functioning as time steps, the second is over all structures in h(i) and the third over all possible utility functions. 8 So, our formalization of the main imperative of preference utilitarianism turns out to be nothing more than maximizing global, all-time expected utility (of every space-time embedded agent that ever comes into existence). In general, the value of the series depends on the order of these infinite sums. 9 Also, the series can diverge. Nonetheless it may still be usable for comparing histories in many cases. 10
 Related work Preferentist utilitarianism has become a common form of utilitarianism in the second half of the 20th century, with the best known proponents being Hare and Singer. However, the intuitions underlying the presented formalization are different from the most common ethical intuitions in preference utilitarianism. Since our formal preference utilitarianism is not meant to describe a decision procedure for humans (or, more generally and in  Hare's (1981, pp. 44f.)  terminology, non-"archangels"), we do not consider an application-oriented utilitarianism like Hare's two-level consequentialism.  (Hare 1981, p. 25ff .) Also, most preference utilitarians ascribe preferences only to humans (or abstract agents) and do not contain prioritization among individuals,  (Harsanyi 1982, p. 46)  or they use a low number of classes of moral standing.  (Singer 1993, pp. 101ff., 283f.)  Whereas some have pointed out that a variety of behavior and even trivial systems can be viewed from an "intentional stance",  (Dennett 1971 (Dennett , 1989 compare Hofstadter 2007, pp. 52ff.)  only relatively recent articles in preference utilitarianism have discussed the connection between goal-directed behavior and ethically relevant preferences and with the universality of the former pointed out the potential universality of the latter.  (Tomasik 2015b, ch. 7; Tomasik 2015a, ch. 4, 6; Tomasik 2015c)  This idea is an important step when formalizing preference utilitarianism because otherwise one would have to define moral standing depending on other, usually binary, notions: being alive, the ability to suffer (Bentham 1823, ch. 17 note 122) personhood  (Gruen 2014, ch. 1) , free will, sentience and (self-)consciousness  (Singer 1993, pp. 101ff.)  or the ability of moral judgment. However, all of them seem to be very difficult to define (universally) in physical systems in the intended binary sense. 11 Also, continuous definitions of these terms are often connected with goal-directed behavior.  (Tomasik 2015a, ch. 4; Wolfram 2002 Wolfram , p. 1136  9 Specifically, the Riemann series theorem states that any conditionally convergent series can be reordered to have arbitrary values.  10  It is very important to differentiate the series from its value. Otherwise, one may identify the series with positive or negative infinity or as being undefined. Two infinite values of the series would then not be comparable anymore, which  Bostrom (2011)  identified as a problem for (consequentialist) ethics. But this problem can sometimes be eliminated by comparing the series itself to another. In this particular case, a history h is better than another history h , if i Spc u u(h)P(u|(h(i)| Spc )@i) − u(h )P(u|(h (i)| Spc )@i) > 0, where h(i)| Spc : Spc → S : c → h(i)(c) denotes the restriction of h(i) to Spc, i.e. the structure on Spc in the state h(i). If no such relation can be established then the two histories are arguably incomparable or may be called approximately equally good. Again, the ordering could be important in some cases, see footnote 9.
 Fig. 3 Comparison between formalizations of utilitarianism. The first row shows the formalization of this paper, the second row is adapted from  Harsanyi (1982, p. 46) , and the third row from  Gips (2011, p. 245) . The utility of an agent n is denoted by U n and its weight by w n sum over all agents utility of an agent i str @i u P (u|str @i)u(h) agent n∈N U n agent n∈N w n • U n Whereas most ethical work is conducted informally,  (McLaren 2011, p. 297; Gips 2011, p. 251)  there has been some formal work at the intersection of (utilitarian) ethics, game theory and economics, most notably by  Harsanyi (1982) . Some formalization has also been conducted in the realm of machine ethics.  (Anderson et al. 2004; Gips 2011, pp. 245ff .) However, influenced by game theory and dualist traditions in philosophy, they are based on the classic agent-environment-model as displayed in Fig.  1  and assume utility functions (or even the utilities in different trajectories themselves) as given by the world model. Nonetheless, there is at least one parallel: all models of utilitarianism contain the notion of summing the utility over all agents. As shown in Fig.  3 , both the definition of all agents and how to obtain the utility or welfare of an agent differ among formalizations. In Artificial Intelligence, the idea of learning preferences has become more popular, e.g. see Fürnkranz and Hüllermeier (2010) and  Nielsen and Jensen (2004)  for technical treatments or  Bostrom (2014, pp. 192ff.)  for an introduction in the context of making an AI do what the engineers value. However, most of the time, the agent is still presumed to be separated from the environment. Nonetheless, the idea of evaluating space-time-embedded intelligence is beginning to be established in artificial (general) intelligence,  (Orseau and Ring 2012)  which is closely related to the probability distribution P(str@i|u).
 Conclusion By reversing Dennett's intentional stance with Bayes' theorem, we were able to ascribe preferences to physical objects and thus formalize preference utilitarianism in cellular automata. Theoretically, such formalizations can function as a specification for an artificial intelligence or more generally as a basis for "paradise engineering" (e.g. see  Ettinger 2009, p. 124) . However, there are several potential problems that require further work before such practical applications of our formalization or improved variations of it can be approached: -Through sums over all structures, possible utility functions and states and the application of incomputable concepts like Solomonoff's prior in P(u), our formalization is incomputable in theory and practice. So even in simulations of cellular automata our formalization is not immediately applicable. -Computing our global welfare function in the real world is even more difficult, because it requires full information about the world on particle level. Also, the formalization must first be translated into the physical laws of our universe. -The difficulty to apply our formalization is by no means only relevant to actually using it as a moral imperative. Instead, it is also relevant to discussing our formalization from a normative standpoint: Even though the derivation of our formalization is plausible, it may still differ significantly from intuition. There could be some kind of trivial agents with trivial preferences that dominate comparison of different histories. Because the formalization's incomputability makes it difficult to assess whether such problems are present, further work on its potential flaws is necessary. Based on such discussion, our formalization may be revised or even discarded. In any case, we could learn a lot from its shortcomings especially due to the formalization's simplicity and plausible derivation. -We outlined how P(str@i|u) and P(u) could be determined in principle. However, they need to be specified more formally, which in the case of P(str@i|u) seems to require a solution to the problem of normative decision theory. Some problems of our formalization could inspire additional refinements of these distributions. Fig. 1 1 Fig. 1 The classic agent-environment-model
 Fig. 2 2 Fig. 2 A state of a two-dimensional cellular automaton. It is very unclear, what agents are and which preferences they have. Adapted from http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life#mediaviewer/ File:Conways_game_of_life_breeder
			 For introductions to and ethical discussions of the underlying notion of preference utilitarianism see Tomasik (2015a, b) .
			 The choice of deterministic systems was made primarily to simplify the formalization. It appears to be unproblematic to transfer formal preference utilitarianism to non-deterministic systems, but defining non-deterministic cellular automata themselves is a little more difficult.
			 Other codomains of utility functions seem possible as long as they are subsets of a totally ordered vector space over R. Intervals like [0, 1] seem specifically suitable, because they avoid problems of infinite utility and allow for normalization. (Isbell 1959 )4  Alternatively, one can try to avoid this hypothetical experiment by predicting the organism's behavior. For example, one could try to ask the organism what it would do or infer its typical behavior from its internals.5  Including i into the data is important, because otherwise identical structures at different points in time would have identical utility functions. This is a problem, when the utility function u is applied to the whole
			 If Solomonoff's prior is chosen for P(u), all incomputable utility functions have zero probability. Since the set of computable functions is countable, only countably many summands could possibly be non-zero. 8 More precisely, but less elegantly, one could write∞ i=0 Spc∈Fin(C) u:(S C ) N →R u(h)P(u|(h(i)| Spc )@i),where Fin(C) := {A ⊆ C||A| ∈ N} is the set of finite subsets of C and h(i)| Spc : Spc → S : c → h(i)(c) is the restriction of the state h(i) to the space Spc and therefore the structure on that space.
			 For example, Wolfram (2002 Wolfram ( , pp. 823-825, 1178Wolfram ( -1180 and Emmeche (1997)  discuss the property of life,Hofstadter (2007, pp. 9-24, 51-54)  discusses consciousness andArneson (1998, p. 5) discusses personhood.