Abstract: This paper focuses on the fact that AI is predominantly portrayed as white-in colour, ethnicity, or both. We first illustrate the prevalent Whiteness of real and imagined intelligent machines in four categories: humanoid robots, chatbots and virtual assistants, stock images of AI, and portrayals of AI in film and television. We then offer three interpretations of the Whiteness of AI, drawing on critical race theory, particularly the idea of the White racial frame. First, we examine the extent to which this Whiteness might simply reflect the predominantly White milieus from which these artefacts arise. Second, we argue that to imagine machines that are intelligent, professional, or powerful is to imagine White machines because the White racial frame ascribes these attributes predominantly to White people. Third, we argue that AI racialised as White allows for a full erasure of people of colour from the White utopian imaginary. Finally, we examine potential consequences of the racialisation of AI, arguing it could exacerbate bias and misdirect concern.

The Whiteness of AI

Introduction It is a truth little acknowledged that a machine in possession of intelligence must be white. Typing terms like "robot" or "artificial intelligence" into a search engine will yield a preponderance of stock images of white plastic humanoids. Perhaps more notable still, these machines are not only white in colour, but the more human they are made to look, the more their features are made ethnically White.  1  In this paper, we problematize the often unnoticed and unremarked-upon fact that intelligent machines are predominantly conceived and portrayed as White. We argue that this Whiteness both illuminates particularities of what (Anglophone Western) society hopes for and fears from these machines, and situates these affects within long-standing ideological structures that relate race and technology. Race and technology are two of the most powerful and important categories for understanding the world as it has developed since at least the early modern period. Yet, as a number of scholars have noted, their profound entanglement remains understudied  (Sinclair 2004; de la Peña 2010) . There are a number of possible reasons for this-and, as Bruce Sinclair writes, "racial prejudice dominates all of them"  (Sinclair 2004, 1) . They include the lack of first-or secondhand accounts of the role of people of colour in the development and use of technology; persistent stereotypes about technology as the province and product of one particular racial group-White people; and the persistent tendency of members of that group, who dominate the academy in the US and Europe, to refuse to see themselves as racialised or race as a matter of concern at all. This lack of scholarly attention is surprising because, as Michael Adas elucidated in 1989, the idea of technological superiority was essential to the logic of colonialism. Not only was superior weaponry and transportation (etc.) necessary for large-scale conquest and control of foreign territory, it was also part of its justification: proof that White Europeans were an advanced civilisation with a right to rule over others  (Adas 1989) . Fortunately, this lack of attention is increasingly being remedied, and the relationship between race and technology is beginning to garner the kind of attention that has since the 1970s been given to gender and technology, following the pioneering work of Donna  Haraway, Sandra Harding, and Evelyn Fox Keller (Haraway 1991; Harding 1986; Keller 1985) . This includes attention to this century's ubiquitous digital technologies. In 2006, Lisa Nakamura asked, "How do we make cyberculture studies a field that as a matter of course employs critical race theory and theories of cultural difference…?"  (Nakamura 2006, 35) . Since then, a number of significant works have attempted to do just that, including Safiya Noble's Algorithms of Oppression and Ruha Benjamin's Race After Technology  (Noble 2018; Benjamin 2019) . This paper aims to contribute to this body of literature on race and technology by examining how the ideology of race shapes conceptions and portrayals of artificial intelligence (AI). Our approach is grounded in the philosophy of race and critical race theory, particularly the Black feminist theories of bell hooks, Sylvia Wynter and Alexander G.  Weheliye (hooks 1992 Weheliye (hooks 1997 Wynter 2003; Weheliye 2014), and work  in Whiteness studies, including that of Richard Dyer, Joe R. Feagin, and Ruth Frankenberg  (Dyer 1997; Feagin 2013; Frankenberg 1997a) . In 2006, Feagin coined the term "white racial frame" to describe those aspects of the Anglophone Western worldview that perpetuate a racialised hierarchy of power and privilege  (Feagin 2006 ). In his words, "the white racial frame includes a broad and persisting set of racial stereotypes, prejudices, ideologies, interlinked interpretations and narratives, and visual images"  (Feagin 2013, xi) . Although it reached its peak in the age of colonial expansion, this framing persists: "Today, as whites move through their lives, they frequently combine racial stereotypes and biases (a beliefs aspect), racial metaphors and concepts (a deeper cognitive aspect), racialised images (the visual aspect), racialised emotions (feelings), interpretive racial narratives, and inclinations to discriminate within a broad racial framing"  (Feagin 2013, 91) . In essence, this paper examines how representations of AI reflect this White racial frame. One of the main aims of critical race theory in general, and Whiteness studies in particular, is to draw attention to the operation of Whiteness in Western culture. The power of Whiteness's signs and symbols lies to a large extent in their going unnoticed and unquestioned, concealed by the myth of colour-blindness. As scholars such as Jessie Daniels and Safiya Noble have noted, this myth of colour-blindness is particularly prevalent in Silicon Valley and surrounding tech culture, where it serves to inhibit serious interrogation of racial framing  (Daniels 2013 (Daniels , 2015 Noble 2018) . Hence the first step for such an interrogation is, in Richard Dyer's term, to "make strange" this Whiteness, de-normalising and drawing attention to it  (Dyer 1997, 10) . As Steve Garner puts it, the reason "for deploying whiteness as a lens is that it strips a normative privileged identity of its cloak of invisibility"  (Garner 2007, 5) . This is our primary intention in examining intelligent machines through the White racial frame. In the next section of this paper, we first lay out current evidence for the assertion that conceptions and portrayals of AI-both embodied as robots and disembodied-are racialised, then evidence that such machines are predominantly racialised as White. In the third section of the paper, we offer our readings of this Whiteness. Our methods are qualitative. As de la Peña writes: "Studying whiteness means working with evidence more interpretive than tangible; it requires imaginative analyses of language and satisfaction with identifying possible motivations of subjects, rather than definitive trajectories of innovation, production, and consumption" (de la Peña 2010, 926). We offer three interpretations of the Whiteness of AI. First, the normalisation of Whiteness in the Anglophone West can go some way to explaining why that sphere's products, including representations of AI, are White. But we argue that this argument alone is insufficient. Second, we argue that to imagine an intelligent (autonomous, agential, powerful) machine is to imagine a White machine because the White racial frame ascribes these attributes predominantly to White people. Thirdly, we argue that AI racialised as White allows for a full erasure of people of colour from the White utopian imaginary. Such machines are conceived as tools that will replace "dirty, dull, or dangerous" tasks  (Murphy 2000, 16) , including replacing human interactions that are considered metaphorically dirty: White robot servants will allow the White master to live a life of ease unsullied by interaction with people of other races.
 Seeing the Whiteness of AI Our concern in this paper is with the racialisation (as White) of both real and imagined machines that are implied or claimed to be intelligent. By racialisation, we mean the ascription of characteristics that are used to identify and delineate races in a given racial frame, which in this case is the Anglophone West. Feagan notes: Among the most important ingredients of this frame are: (1) the recurring use of certain physical characteristics, such as skin colour and facial features, to differentiate social groups; (2) the constant linking of physical characteristics to cultural characteristics; and (3) the regular use of physical and linked cultural distinctions to differentiate socially "superior" and "inferior" groups in a social hierarchy  (Feagin 2013, 41) . It is worth noting that "physical characteristics" need not only refer to those that are visible: voice and accent are also used as markers for social categorisation. Similarly, the category "cultural characteristics" is also used expansively and can include markers such as dialect, mannerisms, and dress codes, as well as mental and moral qualities, such as diligence, industriousness, reliability, trustworthiness, inventiveness, and intellectual ability. Indeed, these mental and moral qualities have always been an essential part of the racial frame, as it is largely on the basis of these that claims of superiority or inferiority have been made.
 Machines Can Be Racialised That machines can be racialised, in the sense that they can be given attributes that enable their identification with human racial categories, has been empirically demonstrated. For example, in one study, Christoph Bartneck and colleagues took pictures of the humanoid Nao robot and adjusted the colouration to match the skin tone of stock images of White and Black people  (Bartneck et al. 2018) . They then asked participants to define the race of the robot with several options including "does not apply". A minority-ranging across the experiments from 7 to 20%-chose the "does not apply" option, while a majority-ranging from 53 to 70%-identified the robots as belonging to the race from which their colouration derived. They concluded "Participants were able to easily and confidently identify the race of robots according to their racialization [...] Thus, there is also a clear sense in which these robotsand by extension other humanoid robotsdo have race"  (Bartneck et al. 2018, 201) . This should not be surprising. Many machines are anthropomorphised-that is, made to be human-like to some degree-in order to facilitate human-machine interaction. This might involve obvious physical features (a head on top, two eyes, a mouth, four limbs, bipedalism, etc.), but it can also include invisible features such as a humanlike voice, or human-like interactions, such as politeness or humour. Given the prevalence of racial framing, in most contexts, to be human-like means to have race. Consequently, as Liao and He point out in their discussion of the racialisation of psychotherapeutic chatbots, "racial identity is an integral part of anthropomorphized agents"  (Liao and He 2020, 2) . They go on to explore a number of racial cues for virtual agents, including visual cues such as skin colour, but also cultural signifiers such as names (e.g. for male names, Jake as White, Darnell as Black, and Antonio as Hispanic). Similarly, "even text-based conversational exchanges"-that is, those with no visual component at all-"perform a racial or ethnic identity" through the interlocutors' choice of dialect, etc.  (Marino 2014, 3) . Given the sociopolitical importance of the racial frame in structuring people's interactions, if machines are really being racialised, then we would expect this to have an impact on how people interact with these machines. Numerous studies show just this. For example, Liao and He found that a person's "perceived interpersonal closeness" with a virtual agent is higher when the virtual agent has the same racial identity as that person  (Liao and He 2020, 2) . Other studies reflect the extent to which racism-prejudicial treatment on the basis of race-is intrinsic to racial framing. As detailed in their paper "Robots Racialized in the Likeness of Marginalized Social Identities are Subject to Greater Dehumanization than Those Racialized as White", Strait et al. analysed free-form online responses to three videos, each depicting a female-gendered android with a different racial identity: Black, White, and East Asian. Their aim was to assess whether the same kind of marginalising and dehumanising commentary that is applied to real people of colour would be applied to these robots. They found that the valence of the commentary was significantly more negative towards the Black robot than towards the White or Asian ones and that both the Asian and Black robots were subject to over twice as many dehumanising comments as the White robot  (Strait et al. 2018) . Two recent studies have further examined the transfer of bias to machines using the "shooter bias" paradigm. This paradigm was first described in the 2002 paper "The Police Officer's Dilemma: Using Ethnicity to Disambiguate Potentially Threatening Individuals"  (Correll et al. 2002) . It used a simple video game featuring images of (real) Black and White male targets, each holding either a gun or a nonthreatening object. Participants were instructed to shoot only armed targets. A clear racial bias was identified: "participants fired on an armed target more quickly when he was African American than when he was White, and decided not to shoot an unarmed target more quickly when he was White than when he was African American"  (Correll et al. 2002 (Correll et al. , 1325 . Studies by  Bartneck et al. and Addison et al. used  the same methodology to examine whether this "shooter bias" would be transferred to racialised robots  (Bartneck et al. 2018; Addison et al. 2019) . They found that "people showed a similar shooter bias toward robots racialized as Black relative to White in a similar fashion as they showed toward Black vs. White humans, no matter their own race"  (Addison et al. 2019, 493) .
 Whiteness as the Norm for Intelligent Machines The previous section shows that research has empirically demonstrated that machines can be racialised and that this racialisation includes transfer of the attendant biases found in the human world. In this subsection, we will survey evidence for the extent to which AI systems-machines purported to be intelligent-are predominantly racialised as White. We will look briefly at four categories: real humanoid robots, virtual personal assistants, stock images of AI, and portrayals of AI in film and television.
 The Whiteness of Humanoid Robots A number of commentators have remarked on the preponderant Whiteness of humanoid robots. In their proposed "code of ethics" for human-robot interaction Riek and Howard note the "lack of diversity in robot morphology and behavior": In terms of race, with precious few exceptions, such as Hanson's Bina48, the vast majority of android and gynoid robots are Asian or Caucasian in their features for no discernible reason. Furthermore, most of these robots tend to have a eurocentric design with regards to their appearance, behavior, and voice.  (Riek and Howard 2014, 4)  Human-computer interaction researchers Christoph Bartneck and colleagues, who conducted some of the studies cited above, have also noted that robots are usually racialised as White: "most of the main research platforms for social robotics, including Nao, Pepper, and PR2, are stylized with white materials and are presumably White"  (Bartneck et al. 2018, 202) . Finally, media studies and literary scholar Jennifer Rhee notes the "normalization and universalization of whiteness" as expressed both in earlier robotics research and in robot toys: "Kismet, with its blue eyes, light brown eyebrows, and pink ears, also 'normalizes whiteness', as do other robot companions, such as the blonde-haired, blue-eyed Cindy Smart Doll and the similarly blonde-haired, blue-eyed My Friend Cayla."  (Rhee 2018, 105) . Although robots such as Nao and Pepper have enjoyed commercial success, neither has received quite the attention garnered by Sophia from Hanson Robotics. This machine consists foremost of a White humanoid head, sometimes also with an upper torso (see Fig.  1 ). It has not only given numerous high-profile television interviews but also received political honours, including in 2017 receiving citizenship of Saudi Arabia and becoming an "Innovation Champion" for the United Nations Development Programme  (Weller 2017 ; UNDP 2017).
 The Whiteness of Chatbots and Virtual Assistants Though conversational agents do not exhibit any visual racial cues, they are racialised by means of sociolinguistic markers  (Sweeney 2016; Villa-Nicholas and Sweeney 2019) . Discussing ELIZA, an influential natural language processing program created by Joseph Weizenbaum at the MIT AI Laboratory in 1966, Mark Marino writes: "If ELIZA presented a bot that tried to imitate language, it was performing standard white middle-class English, without a specific identifying cultural inflection... language without culture, disembodied, hegemonic, and, in a word, white"  (Marino 2014, 5) . Since then, natural language processing has entered the mainstream, with "virtual assistants" existing in many people's pockets, handbags, or homes through devices such as smartphones. Indeed, this is one of the most common ways in which people interact with technology that could be labelled "AI". These tools present their designers with many decisions about socio-cultural positioning. Ruha Benjamin recalls this anecdote: A former Apple employee who noted that he was "not Black or Hispanic" described his experience on a team that was developing speech recognition for Siri, the virtual assistant program. As they worked on different English dialects -Australian, Singaporean and Indian Englishhe asked his boss: "What about African American English?" To this his boss responded: "Well, Apple products are for the premium market."  (Benjamin 2019, 28)  As a further example, she describes a Black computer scientist who chose a White voice for his app rather than a Black one, so as not to "create friction"  (Benjamin 2019, 28-29) . So while some designers might be unconsciously racialising their products as White, others are doing so in full awareness of this choice.
 The Whiteness of Stock Images of AI As anyone working in the field will know, stock images of AI, at least when anthropomorphised, are overwhelmingly white and arguably overwhelmingly White. The more realistically humanoid these machines become, the more Caucasian in their features. Such images are used to illustrate not only generalist newspaper articles and corporate slideshows but also specialist and technical works, and even works of a critical nature, such as Harry Collins's Artifictional Intelligence (Polity, 2018) and Anthony Elliott's The Culture of AI (Routledge, 2018) (Fig.  2 ). The prevalence of such images is reflected in the results of search engines. Such searches are a useful indicator of how a subject is portrayed at a given time, for two reasons. First, search engines are very widely used (approximately 3.5 billion searches are made on Google every day, or 40 thousand per second 2 ) and can therefore be considered a highly influential source of information and perceptions. Second, the nature of such search engines means that they are not only promoting certain ideas and perceptions but also reflecting their existing prevalence. While the exact nature of Google's search, ranking, and result presentation algorithms is proprietary, we know that they evaluate (crudely put) influence and popularity-for example, in terms of how many other sites link to a given website. So the fact that certain images are shown when someone searches for a relevant term means not only that those images are being thus promoted by some of the most powerful organs of content mediation in existence today but also that these images are already widespread and used on other influential websites, as that is what underlies their promotion by the search engines. Consequently, search results are increasingly examined by scholars, including in the study of racial bias. For example, in her 2018 book Algorithms of Oppression: How Search Engines Reinforce Racism, Safiya U. Noble identifies many ways in which such sites reflect and exacerbate prejudice, such as the search results for "Latinas" that feature mostly porn  (Noble 2018, 75, 155)  or the White men who come up when searching for images of professions such as "construction worker", "doctor", or "scientist"  (Noble 2018, 82-83) . In order to get an indication of the prevalence of these racialised machines on the internet, we conducted two image searches on Google (the most widely used search engine) using the anonymous Tor browser to ensure results were not influenced by our personal search histories and locations. We first searched on the term "artificial intelligence": the top results are in Fig.  3 . Some of these results are too abstract, featuring stylised brains and circuits, for example, to be considered racialised. However, among the results showing humanoid figures, racialisation as White predominates. First, two pictures show actual human hands, and both are White. Second, a further two pictures show humanoid robots, and both are White and could thus be read as White, as  Bartneck et al. suggest (Bartneck et al. 2018, 202) . Therefore, we might say that inasmuch as the machines are racialised, they are racialised as White. In order to focus more on representations of embodied, anthropomorphic AI, we also searched for "artificial intelligence robot": the top results are in Fig.  4 . As is clear, this search produces an even greater preponderance of images that are either white in colour or racialised as White or both.
 The Whiteness of AI in Film and Television These contemporary stock images distil the visualisations of intelligent machines in Western popular culture as it has developed over decades. In science fiction from the nineteenth century onwards, AI is predominantly imagined as White. For example, the Terminator (Arnold Schwarzenegger), RoboCop (Peter Weller and Joel Kinnaman), all of the "replicants" in the Blade Runner franchise (e.g. Rutger Hauer, Sean Young, and Mackenzie Davis), Sonny in I, Robot (Alan Tudyk), Ava in Ex Machina (Alicia The Whiteness of AI Vikander) (Fig.  5 ), and Maria in Metropolis (Brigitte Helm) are all played by White actors and are visibly White on screen. Androids made of metal or plastic are also usually given White facial features, such as the robots in the 2007 film I, Robot. Even disembodied AI is imagined as White: HAL-9000 in 2001: A Space Odyssey and Samantha in Her are voiced by White actors. All of these AIs come from Hollywood films; they have been produced in a country in which 18% of the population is Hispanic, but in which only one fictional robot has that background: Bender Rodríguez in the animated TV series Futurama, who is canonically constructed in Mexico-but who is voiced by the White voice actor John DiMaggio. Only very recent TV shows with a large cast of androids, such as Westworld and Humans, have attempted to address this with AI characters evincing a mix of skin tones and ethnicities. This preponderance of intelligent machines racialised as White led Dyer to posit "the android as a definition of whiteness"  (Dyer 1997, 213) .
 Understanding the Whiteness of AI We offer three interpretations of the racialisation of intelligent machines as White: the Whiteness of their creators perpetuating itself; the Whiteness of the attributes ascribed to AI; and the extent to which AI permits the erasure of people of colour from the White utopia.
 Whiteness Reproducing Whiteness In European and North American societies, Whiteness is normalised to an extent that renders it largely invisible. As Toby Ganley puts it in his survey of Whiteness studies, "the monopoly that whiteness has over the norm" is one of the field's two unifying insights-the other being that it confers power and privilege  (Ganley 2003, 12) . Richard Dyer describes this as the view that "other people are raced, we are just people"  (Dyer 1997, 1) . This normalisation means that Whiteness is not perceived by majority populations as a distinct colour, but rather as an absence of colour-colour Fig.  5  Alicia Vikander as Ava in Ex Machina. Source: Youtube both in the literal sense and in the sense of race. Consequently, the Whiteness of AI could be considered simply a default. It does not appear as a feature, but is transparent, like the air we breathe: the "unmarked marker", as Ruth Frankenberg calls it  (Frankenberg 1997b, 1) . The majority of White viewers are unlikely to see humanlike machines as racialised at all, but simply as conforming to their idea of what "human-like" means. For non-White people, on the other hand, Whiteness is never invisible in this manner, as bell hooks reminds us  (hooks <PHONE>) . So-called colour-blindness, an attitude of not seeing race, and of presuming that people in contemporary society are no longer disadvantaged on the basis of race, is itself a narrative that perpetuates White hegemony: "communities of color frequently see and name whiteness clearly and critically, in periods when white folks have asserted their own 'color blindness'"  (Frankenberg 1997b, 4) . Noble argues that "central to these 'colorblind' ideologies is a focus on the inappropriateness of 'seeing race'"-a view that she argues is dominant among Silicon Valley technologists, who "revel in their embrace of colorblindness as if it is an asset and not a proven liability"  (Noble 2018, 168) . Such colour-blindness is a liability because it obscures the normalisation of Whiteness and marginalisation of other racialised groups-and the real world effects this has, such as facial recognition technologies not distinguishing Black or East Asian faces  (Buolamwini and Gebru 2018) . Given the normalisation of Whiteness, for some designers, to make a human-like machine will unthinkingly mean to make a White machine. As Dyer puts it: "white people create the dominant images of the world and don't quite see that they thus create the dominant images of the world in their own image"  (Dyer 1997, 9) . But this alone is not a satisfactory explanation of the Whiteness of AI, as not all entities-more specifically, not all intelligent, humanoid entities-imagined by predominantly White industries are portrayed as White. For example, Western science fiction has a long tradition of White authors racialising extraterrestrials as non-White. In the late nineteenth century, for instance, the real-world fear of the "Yellow Peril" was metaphorically addressed in science fiction by racialising extraterrestrial invaders as East Asian. The Flash Gordon franchise gained its lead villain in a 1934 comic, which introduced the tyrannical emperor of the planet Mongo-the Orientalised alien Ming the Merciless. Such is the villain in Flash Gordon -a trident bearded, slanty eyed, shiny doomed [sic], pointy nailed, arching eyebrowed, exotically garbed Oriental named Ming, who personifies unadulterated evil. A heavy like Ming is not contrived in a comic strip writer's imagination during a coffee break, but rather is the product of perhaps the richest and longest tradition of all of Hollywood ethnic stereotypes.  (Barshay 1974, 24-26)  Dyer points out that Blade Runner similarly deliberately uses East Asian characters in order to offset the whiteness of its protagonists, including the White androids: "the yellow human background emphasises the chief protagonists' whiteness. The whitest of hue are the replicants"  (Dyer 1997, 214) . Racial stereotyping of aliens is not a phenomenon limited to past centuries. The Star Wars prequel trilogy  (Lucas 1999; 2002; 2005)  has been criticised for the "transparent racism" in its depiction of the alien Jar Jar Binks as a West Indian caricature  (Lavender 2011, 193)  reminiscent of blackface minstrelsy  (Williams 1999) , and of the slave trader Watto, an antisemitic Jewish caricature with a large nose, skullcap, Yiddish accent, and obsession with money  (Freedman 2019) . This racialisation of aliens in SF suggests that the racialisation of artificial intelligence is a choice. The White racial frame as perpetuated by the White creators of these works portrays dangerous invaders from another planet as East Asian and bumbling alien petty-criminals as Afro-Caribbean. Therefore, the fact that it portrays AI as overwhelmingly White requires further explanation. In the following sections, we offer two.
 AI and the Attributes of Whiteness While Whiteness functions in part through its invisibility in mainstream discourse, this does not mean it has no distinguishable features of its own. Indeed, the White racial frame has a long history of ascribing certain attributes to Whites and disputing them in others: these are the very claims that have been used to justify colonialism, segregation, and other modes of oppression. We argue that AI is predominantly racialised as White because it is deemed to possess attributes that this frame imputes to White people. We examine these attributes under three key headings: intelligence, professionalism, and power. First, the primary attribute being projected onto these machines is, as the term "AI" suggests, intelligence. Throughout the history of Western thought, but in particular since the seventeenth century in Europe and the territories it colonised, intelligence has been associated with some humans more than others  (Carson 2006) . The idea that some races were more mentally able than others was crucial to the legitimation of the advancing colonial project. Those deemed less intelligent-in the words of Rudyard Kipling, "Half-devil and half-child"-were judged unqualified to rule themselves and their lands. It was therefore legitimate-even a duty, "the white man's burden" as Kipling put it-to destroy their cultures and take their territories  (Kipling 1899) . Through the nineteenth century, strenuous efforts were made to empirically demonstrate and measure this intellectual difference, culminating in the development of the IQ test  (Gould 1981) . Although explicit associations between racial groups and intelligence declined after the Second World War, (a) they continue to be made in right-wing circles  (Saini 2019 ) and (b) implicit or unconscious associations between race and intelligence persist widely (see, for example, van den  Bergh et al. 2010; Okeke et al. 2009) . Given the White racial frame has for centuries promoted the association of intelligence with the White, European race, it is to be expected that when this culture is asked to imagine an intelligent machine, it imagines a White machine. A crucial aspect of the idea of intelligence is generality. Intelligence is often defined as a "general mental capability"  (Gottfredson 1997) , and in AI, the concept of "artificial general intelligence"-a system with the kind of flexible mental capabilities humans have-is often considered to be the original and primary goal of the field  (Crevier 1993) . But in the White racial frame, not all humans are considered to have this attribute to the same degree. As Weheliye puts it, using Sylvia Wynter's idea of "the Man"-the Enlightenment, Western, White male subject, "In the context of the secular human, black subjects, along with indigenous populations, the colonised, the insane, the poor, the disabled, and so on as limit cases by which Man can demarcate himself as the universal human"  (Weheliye 2014, 24) . According to the White racial frame, it is the rational, scientific thought of the White Westerner that lays claim to universal validity-or, we might say, true generality. Other races, by contrast, are framed as particular and subjective, constrained by the limits of their non-ideal bodies and cultures to think thoughts that are partial and parochial. To imagine a truly intelligent machine, one with general intelligence is therefore to imagine a White machine. Second, much of the current discourse around AI focuses on how it is, or will soon be, capable of professional work. This is frequently claimed to be what makes the present wave of automation different from previous waves, in which machines became capable of supplanting manual and semi-skilled labour  (Ford 2015) . Professional work-law, medicine, business, and so forth-is at the upper end of pay and status scales. White Europeans and North Americans have historically not considered all humans equally fit for such roles and have kept them closed to people who lacked the requisite connections, wealth, or other in-group identifiers. Universities, the gateways to the professions, have long histories of excluding people of colour from their ranks  (Burrow 2008, 107) . The historic exclusion of anyone other than White men shapes to this day what mainstream White culture imagines when imagining someone fulfilling such roles. Safiya Noble shows that it took years of criticism before search engines adjusted their algorithms so that searching for "engineer" or "doctor" stopped exclusively returning images of White men  (Noble 2018) . But the underlying bias, on which the algorithms fed, remains. To imagine a machine in a white-collar job is therefore to imagine a White machine. Third, hierarchies of intelligence and of professional status are of course also hierarchies of power. Consequently, power relations are implicit in the previous two categories. However, it is worth also considering power separately, because power struggles between AI and humans are such a common narrative trope. Alongside the narrative that robots will make humans redundant, an equally well-known narrative is that they will rise up and conquer us altogether  (Cave and Dihal 2019) . These are both narratives about machines becoming superior to humans: stories in which they become better at every task, leaving humans with nothing to do, from E.M. Forster's 1909 short story 'The Machine Stops' to the Oscar-winning film WALL-E, or in which they outwit and subjugate those who built them, as in the Terminator film franchise or the film Ex Machina  (Forster 1909; Stanton 2008; Cameron 1984; Garland 2015) . When White people imagine being overtaken by superior beings, those beings do not resemble those races they have framed as inferior. It is unimaginable to a White audience that they will be surpassed by machines that are Black. Rather, it is by superlatives of themselves: hyper-masculine White men like Arnold Schwarzenegger as the Terminator, or hyperfeminine White women like Alicia Vikander as Ava in Ex Machina. This is why even narratives of an AI uprising that are clearly modelled on stories of slave rebellions depict the rebelling AIs as White-for example, in Blade Runner  (Dihal 2020) . The implication of this racialisation is that these machines might genuinely be superior, or are at least worthy adversaries. The use of White bodybuilders such as Arnold Schwarzenegger to play the evil robots suggests this. As Dyer points out, Schwarzenegger's physique suggests "the body made possible by [...] natural mental superiority. The point after all is it built, a product of the application of thought and planning, an achievement"  (Dyer 1997, 164) . Consequently, for a White technologist or author, to imagine a superior anthropomorphic machine is to imagine a White machine. In summary, popular conceptions of AI suggest these machines have general intelligence, are capable of professional jobs, and/or are poised to surpass and supplant humanity. In the White imagination, such qualities are strongly associated with Whiteness. It is no surprise, therefore, that in mainstream Western media, such machines are portrayed as White.
 White Utopia While we believe the attribution to AI of these qualities, so strongly associated with Whiteness, goes a long way to making sense of the racialisation of anthropomorphic intelligent machines, we also want to propose one further hypothesis: that the Whiteness of the machines allows the White utopian imagination to fully exclude people of colour. One of the most pertinent hopes for artificial intelligence is that it will lead to a life of ease  (Cave and Dihal 2019) . As a tool that can take over "dirty, dull, or dangerous" jobs, it relieves its owners from work they do not want to do, enabling them to pursue leisure. As critical race theorists have repeatedly pointed out, the leisure currently available to the wealthier classes is disproportionately facilitated by the labour of working-class women of colour  (hooks 1992 Rhee 2018) . bell hooks shows that the people performing this labour are actively kept invisible, even when the White master and the coloured servant are physically present in the same space. She cites the memoirs of a White heiress who grew up with Black servants in her house: "Blacks, I realized, were simply invisible to most white people, except as a pair of hands offering a drink on a silver tray"  (hooks <PHONE>, 168) . As this forced pretence of invisibility shows, interactions with non-White servants are undesirable to the White master: such interactions are almost literally considered a "dirty job". Depictions of people of colour as being dirty and unwashed, eating dirty food, living in the dirt, even of being the colour of excrement have contributed to the development of both the fear of pollution in interactions with people of colour, and the association of Whiteness with cleanliness and purity  (Dyer 1997, 75-76) . This association has been exacerbated by a long history of propaganda preceding conquest and genocide that portrays the racial other as evoking disgust: as vectors of disease, such as lice or rats, or as a literal plague  (Glover 1999, chap. 35; Rector 2014, chap. 3) . The utopia of the White racial frame would therefore rather remove people of colour altogether, even in the form of servants. From the inception of the academic study of science fiction onwards, many critics have pointed out that utopias throughout literary history have been construed on exclusionary, colonialist, and eugenicist premises  (Suvin 1979 (Suvin 2016 Jameson 2005, 205; Ginway 2016, 132) . In Astrofuturism, De Witt Douglas Kilgore shows that mid-twentieth-century American visions of space age utopias are "idealisations ... based on a series of exclusions" (Kilgore 2010, 10): rather than depicting a post-racial or colourblind future, the authors of these utopias simply omit people of colour. AI offers the possibility of making such utopias real. By virtue of its generality, it is imagined as able replace and any unwanted labour-social and cognitive as well as physical  (Cave and Dihal 2019) , so obviating the need for people of colour in any role. Consequently, as Jennifer Rhee points out, advertisements for real AI such as household robots "are striking in their whiteness": they are aimed at showing white middle-class families an ideal leisurely lifestyle. In doing so, she argues, "the images reserve the luxury of liberation from domestic labor for white women, while erasing the women of color who perform this labor, both within their own homes and in the homes of others"  (Rhee 2018, 94) . In some cases, the unsulliedness of this utopia can extend further to exclude all women. Just as people of colour can be associated with offensive physicality, so can women in general, particularly with respect to their reproductive organs. The necessity of sexual intercourse, pregnancy, and childbearing for the continuation of a race that prides itself on rationality and the ability to transcend its physicality is an offensive hurdle that has been imagined as transcendable by science for centuries. As Dyer points out, in the ideology of Whiteness, the elevation of mental over physical prowess has simultaneously been the White race's most valuable achievement and a threat to its own continuation  (Dyer 1997, 27) . It has led to the paradox known as the "White Crisis", in which the White race is seen as under threat of being overwhelmed by "inferior" races that are breeding more prolifically. Transhumanism has been envisioned as a solution to this White Crisis  (Ali 2017) . Seen as a form of offspring, artificial intelligence offers a way for the White man to perpetuate his existence in a rationally optimal manner, without the involvement of those he deems inferior.
 Conclusion and Implications Images of AI are not generic representations of human-like machines, but avatars of a particular rank within the hierarchy of the human. These representations of intelligent machines-and our future with them-are refracted through the White racial frame; their Whiteness a proxy for how we perceive their status and potential. This can cause what is sometimes called representational harms  (Blodgett et al. 2020) . We suggest three. First, this racialisation can amplify the very prejudices it reflects. We have argued that intelligent machines are portrayed as White because that is how the mainstream perceives intelligence and related desirable characteristics. But equally, the consistent portrayal of intelligent machines as White itself transmits this association, so sustaining it. As we have argued elsewhere  (Whittlestone et al. 2019) , bias in representations of AI contributes to a vicious cycle of social injustice: the biased representations can influence both aspiring technologists and those in charge of hiring new staff, shaping whom they consider fit for the field  (Cave 2020) . This could contribute to sustaining a racially homogenous workforce, which will continue to produce products, whether real intelligent machines or their representations, that are biased to benefit that group and disadvantage others. Second, the racialisation of these machines places them within an existing hierarchy of the human in a way that could exacerbate real injustice. Portrayals of AI as White situate these machines in a power hierarchy above currently marginalised groups, such as people of colour. These oppressed groups are therefore relegated to an even lower position in the hierarchy: below that of machine. As machines become ever more important in making automated decisions-frequently about marginalised  (Eubanks 2017) -this be consequential. Automation bias-the tendency of people to favour suggestions from automated decision-making systems over those from humans-has already been evidenced  (Goddard et al. 2012) . We might speculate that it will be exacerbated in cases where such systems are racialised White and the humans in question are not. Third, these portrayals could distort our perceptions of the risks and benefits of these machines. For example, they could frame the debate about AI's impact disproportionately around the opportunities and risks posed to White middle-class men  (Cave 2020) . It is already a common narrative that the current wave of automation differs from those of the past in that "impacts from automation have thus far impacted mostly blue-collar employment; the coming wave of innovation threatens to upend white-collar work as well"  (Pew Research Center 2014) . Public interest and policy therefore often focus on white collar professionals, instead of on marginalized groups, which in reality are likely to be worse affected by the impact of AI  (Eubanks 2017; Noble 2018) . In this paper, we have offered three interpretations of the whiteness and Whiteness of representations of AI. All three, and the implications that we posit, need further investigation. This process is part of what can be described as decolonising AI: a process of breaking down the systems of oppression that arose with colonialism and have led to present injustices that AI threatens to perpetuate and exacerbate. Weheliye describes how he "works towards the abolition of Man, and advocates the radical reconstruction and decolonization of what it means to be human"  (Weheliye 2014, 4) . It is in the field of AI that technology is most clearly entwined with notions of "what it means to be human", both in reality and in cultural fantasies. We hope to have taken a step towards this reconstruction, by drawing attention to the Whiteness of these machines and "making it strange". regulation or exceeds the permitted use, you will need obtain permission directly from the copyright holder. To view a copy of this licence, visit Fig. 2 2 Fig. 2 Covers of Collins 2018, Polity, and Elliott 2018, Routledge
 Fig. 3 3 Fig. 3 Tor browser Google image search result for "artificial intelligence", 13 April 2020

			 Following the increasingly common usage of the capitalised form "Black" to denote the ethnicity and "black" the colour, we use "White" to refer to the ethnicity and "white" the colour.While not yet the norm, as can be seen in our quotations of critics who do not employ this distinction, this usage will make our discussion clearer.
			 The Whiteness of AI
			 https://www.internetlivestats.com/google-search-statistics/ accessed 30 December 2019.Fig. 1 Sophia. Hanson Robotics, April 2020The Whiteness of AI
			 Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


Abstract: For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (

Joshua Achiam
David Held
Aviv Tamar
Pieter Abbeel

Constrained Policy Optimization

Introduction Recently, deep reinforcement learning has enabled neural network policies to achieve state-of-the-art performance on many high-dimensional control tasks, including Atari games (using pixels as inputs)  (Mnih et al., 2015; , robot locomotion and manipulation  (Schulman et al., 2015; Levine et al., 2016; Lillicrap et al., 2016) , and even Go at the human grandmaster level  (Silver et al., 2016) . 1 UC Berkeley 2 OpenAI. Correspondence to: Joshua Achiam <<EMAIL>. Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by the author(s). In reinforcement learning (RL), agents learn to act by trial and error, gradually improving their performance at the task as learning progresses. Recent work in deep RL assumes that agents are free to explore any behavior during learning, so long as it leads to performance improvement. In many realistic domains, however, it may be unacceptable to give an agent complete freedom. Consider, for example, an industrial robot arm learning to assemble a new product in a factory. Some behaviors could cause it to damage itself or the plant around it-or worse, take actions that are harmful to people working nearby. In domains like this, safe exploration for RL agents is important  (Moldovan & Abbeel, 2012; Amodei et al., 2016) . A natural way to incorporate safety is via constraints. A standard and well-studied formulation for reinforcement learning with constraints is the constrained Markov Decision Process (CMDP) framework  (Altman, 1999) , where agents must satisfy constraints on expectations of auxilliary costs. Although optimal policies for finite CMDPs with known models can be obtained by linear programming, methods for high-dimensional control are lacking. Currently, policy search algorithms enjoy state-of-theart performance on high-dimensional control tasks  (Mnih et al., 2016; Duan et al., 2016) . Heuristic algorithms for policy search in CMDPs have been proposed  (Uchibe & Doya, 2007) , and approaches based on primal-dual methods can be shown to converge to constraint-satisfying policies  (Chow et al., 2015) , but there is currently no approach for policy search in continuous CMDPs that guarantees every policy during learning will satisfy constraints. In this work, we propose the first such algorithm, allowing applications to constrained deep RL. Driving our approach is a new theoretical result that bounds the difference between the rewards or costs of two different policies. This result, which is of independent interest, tightens known bounds for policy search using trust regions  (Kakade & Langford, 2002; Pirotta et al., 2013; Schulman et al., 2015) , and provides a tighter connection between the theory and practice of policy search for deep RL. Here, we use this result to derive a policy improvement step that guarantees both an increase in reward and satisfaction of constraints on other costs. This step forms the basis for our algorithm, Constrained Policy Optimization (CPO), which computes an approximation to the theoretically-justified update. In our experiments, we show that CPO can train neural network policies with thousands of parameters on highdimensional simulated robot locomotion tasks to maximize rewards while successfully enforcing constraints.
 Related Work Safety has long been a topic of interest in RL research, and a comprehensive overview of safety in RL was given by  (García & Fernández, 2015) . Safe policy search methods have been proposed in prior work.  Uchibe and Doya (2007)  gave a policy gradient algorithm that uses gradient projection to enforce active constraints, but this approach suffers from an inability to prevent a policy from becoming unsafe in the first place. Bou  Ammar et al. (2015)  propose a theoretically-motivated policy gradient method for lifelong learning with safety constraints, but their method involves an expensive inner loop optimization of a semi-definite program, making it unsuited for the deep RL setting. Their method also assumes that safety constraints are linear in policy parameters, which is limiting.  Chow et al. (2015)  propose a primal-dual subgradient method for risk-constrained reinforcement learning which takes policy gradient steps on an objective that trades off return with risk, while simultaneously learning the trade-off coefficients (dual variables). Some approaches specifically focus on application to the deep RL setting.  Held et al. (2017)  study the problem for robotic manipulation, but the assumptions they make restrict the applicability of their methods.  Lipton et al. (2017)  use an 'intrinsic fear' heuristic, as opposed to constraints, to motivate agents to avoid rare but catastrophic events.  Shalev-Shwartz et al. (2016)  avoid the problem of enforcing constraints on parametrized policies by decomposing 'desires' from trajectory planning; the neural network policy learns desires for behavior, while the trajectory planning algorithm (which is not learned) selects final behavior and enforces safety constraints. In contrast to prior work, our method is the first policy search algorithm for CMDPs that both 1) guarantees constraint satisfaction throughout training, and 2) works for arbitrary policy classes (including neural networks).
 Preliminaries A Markov decision process (MDP) is a tuple, (S, A, R, P, µ), where S is the set of states, A is the set of actions, R : S × A × S → R is the reward function, P : S ×A×S → [0, 1] is the transition probability function (where P (s � |s, a) is the probability of transitioning to state s � given that the previous state was s and the agent took action a in s), and µ : S → [0, 1] is the starting state distribution. A stationary policy π : S → P(A) is a map from states to probability distributions over actions, with π(a|s) denoting the probability of selecting action a in state s. We denote the set of all stationary policies by Π. In reinforcement learning, we aim to select a policy π which maximizes a performance measure, J(π), which is typically taken to be the infinite horizon discounted total return, J(π) . = E τ ∼π [ � ∞ t=0 γ t R(s t , a t , s t+1 )]. Here γ ∈ [0, 1) is the discount factor, τ denotes a trajectory (τ = (s 0 , a 0 , s 1 , ...)), and τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π: s 0 ∼ µ, a t ∼ π(•|s t ), s t+1 ∼ P (•|s t , a t ). Letting R(τ ) denote the discounted return of a trajectory, we express the on-policy value function as V π (s) . = E τ ∼π [R(τ )|s 0 = s] and the on-policy action-value function as Q π (s, a) . = E τ ∼π [R(τ )|s 0 = s, a 0 = a]. The advantage function is A π (s, a) . = Q π (s, a) − V π (s). Also of interest is the discounted future state distribution, d π , defined by d π (s) = (1−γ) � ∞ t=0 γ t P (s t = s|π). It allows us to compactly express the difference in performance between two policies π � , π as J(π � ) − J(π) = 1 1 − γ E s∼d π � a∼π � [A π (s, a)] , (1) where by a ∼ π � , we mean a ∼ π � (•|s), with explicit notation dropped to reduce clutter. For proof of (1), see  (Kakade & Langford, 2002)  or Section 10 in the supplementary material.
 Constrained Markov Decision Processes A constrained Markov decision process (CMDP) is an MDP augmented with constraints that restrict the set of allowable policies for that MDP. Specifically, we augment the MDP with a set C of auxiliary cost functions, C 1 , ..., C m (with each one a function C i : S × A × S → R mapping transition tuples to costs, like the usual reward), and limits d 1 , ..., d m . Let J Ci (π) denote the expected discounted return of policy π with respect to cost function C i : J Ci (π) = E τ ∼π [ � ∞ t=0 γ t C i (s t , a t , s t+1 )]. The set of feasible stationary policies for a CMDP is then Π C . = {π ∈ Π : ∀i, J Ci (π) ≤ d i } , and the reinforcement learning problem in a CMDP is π * = arg max π∈Π C J(π). The choice of optimizing only over stationary policies is justified: it has been shown that the set of all optimal policies for a CMDP includes stationary policies, under mild technical conditions. For a thorough review of CMDPs and CMDP theory, we refer the reader to  (Altman, 1999) . We refer to J Ci as a constraint return, or C i -return for short. Lastly, we define on-policy value functions, actionvalue functions, and advantage functions for the auxiliary costs in analogy to V π , Q π , and A π , with C i replacing R: respectively, we denote these by V π Ci , Q π Ci , and A π Ci .
 Constrained Policy Optimization For large or continuous MDPs, solving for the exact optimal policy is intractable due to the curse of dimensionality  (Sutton & Barto, 1998) . Policy search algorithms approach this problem by searching for the optimal policy within a set Π θ ⊆ Π of parametrized policies with parameters θ (for example, neural networks of a fixed architecture). In local policy search  (Peters & Schaal, 2008) , the policy is iteratively updated by maximizing J(π) over a local neighborhood of the most recent iterate π k : π k+1 = arg max π∈Π θ J(π) s.t. D(π, π k ) ≤ δ, (2) where D is some distance measure, and δ > 0 is a step size. When the objective is estimated by linearizing around π k as J(π k ) + g T (θ − θ k ), g is the policy gradient, and the standard policy gradient update is obtained by choosing  Schulman et al., 2015) . D(π, π k ) = �θ − θ k � 2 ( In local policy search for CMDPs, we additionally require policy iterates to be feasible for the CMDP, so instead of optimizing over Π θ , we optimize over Π θ ∩ Π C : π k+1 = arg max π∈Π θ J(π) s.t. J Ci (π) ≤ d i i = 1, ..., m D(π, π k ) ≤ δ. (3) This update is difficult to implement in practice because it requires evaluation of the constraint functions to determine whether a proposed point π is feasible. When using sampling to compute policy updates, as is typically done in high-dimensional control  (Duan et al., 2016) , this requires off-policy evaluation, which is known to be challenging  (Jiang & Li, 2015) . In this work, we take a different approach, motivated by recent methods for trust region optimization  (Schulman et al., 2015) . We develop a principled approximation to (3) with a particular choice of D, where we replace the objective and constraints with surrogate functions. The surrogates we choose are easy to estimate from samples collected on π k , and are good local approximations for the objective and constraints. Our theoretical analysis shows that for our choices of surrogates, we can bound our update's worst-case performance and worst-case constraint violation with values that depend on a hyperparameter of the algorithm. To prove the performance guarantees associated with our surrogates, we first prove new bounds on the difference in returns (or constraint returns) between two arbitrary stochastic policies in terms of an average divergence between them. We then show how our bounds permit a new analysis of trust region methods in general: specifically, we prove a worst-case performance degradation at each update. We conclude by motivating, presenting, and proving gurantees on our algorithm, Constrained Policy Optimization (CPO), a trust region method for CMDPs.
 Policy Performance Bounds In this section, we present the theoretical foundation for our approach-a new bound on the difference in returns between two arbitrary policies. This result, which is of independent interest, extends the works of  (Kakade & Langford, 2002) ,  (Pirotta et al., 2013) , and  (Schulman et al., 2015) , providing tighter bounds. As we show later, it also relates the theoretical bounds for trust region policy improvement with the actual trust region algorithms that have been demonstrated to be successful in practice  (Duan et al., 2016) . In the context of constrained policy search, we later use our results to propose policy updates that both improve the expected return and satisfy constraints. The following theorem connects the difference in returns (or constraint returns) between two arbitrary policies to an average divergence between them. Theorem 1. For any function f : S → R and any policies π � and π, define δ f (s, a, s � ) . = R(s, a, s � ) + γf (s � ) − f (s), � π � f . = max s |E a∼π � ,s � ∼P [δ f (s, a, s � )]| , L π,f (π � ) . = E s∼d π a∼π s � ∼P �� π � (a|s) π(a|s) − 1 � δ f (s, a, s � ) � , and D ± π,f (π � ) . = L π,f (π � ) 1 − γ ± 2γ� π � f (1 − γ) 2 E s∼d π [D T V (π � ||π)[s]] , where D T V (π � ||π)[s] = (1/2) � a |π � (a|s) − π(a|s) | is the total variational divergence between action distributions at s. The following bounds hold: D + π,f (π � ) ≥ J(π � ) − J(π) ≥ D − π,f (π � ). (4) Furthermore, the bounds are tight (when π � = π, all three expressions are identically zero). Before proceeding, we connect this result to prior work. By bounding the expectation E s∼d π [D T V (π � ||π)[s]] with max s D T V (π � ||π)[s], picking f = V π , and bounding � π � V π to get a second factor of max s D T V (π � ||π)[s], we recover (up to assumption-dependent factors) the bounds given by  Pirotta et al. (2013)  as Corollary 3.6, and by  Schulman et al. (2015)  as Theorem 1a. The choice of f = V π allows a useful form of the lower bound, so we give it as a corollary. Corollary 1. For any policies π � , π, with � π � . = max s |E a∼π � [A π (s, a)]|, the following bound holds: J(π � ) − J(π) ≥ 1 1 − γ E s∼d π a∼π � � A π (s, a) − 2γ� π � 1 − γ D T V (π � ||π)[s] � . ( 5 ) The bound (5) should be compared with equation (  1 ). The term  5 ) is an approximation to J(π � ) − J(π), using the state distribution d π instead of d π � , which is known to equal J(π � ) − J(π) to first order in the parameters of π � on a neighborhood around π  (Kakade & Langford, 2002) . The bound can therefore be viewed as describing the worst-case approximation error, and it justifies using the approximation as a surrogate for J(π � ) − J(π). (1 − γ) −1 E s∼d π ,a∼π � [A π (s, a)] in ( Equivalent expressions for the auxiliary costs, based on the upper bound, also follow immediately; we will later use them to make guarantees for the safety of CPO. Corollary 2. For any policies π � , π, and any cost function C i , with � π � Ci . = max s |E a∼π � [A π Ci (s, a)]|, the following bound holds: J Ci (π � ) − J Ci (π) ≤ 1 1 − γ E s∼d π a∼π � � A π Ci (s, a) + 2γ� π � Ci 1 − γ D T V (π � ||π)[s] � . (6) The bounds we have given so far are in terms of the TV-divergence between policies, but trust region methods constrain the KL-divergence between policies, so bounds that connect performance to the KL-divergence are desirable. We make the connection through Pinsker's inequality  (Csiszar & Körner, 1981) : for arbitrary distributions p, q, the TV-divergence and KL-divergence are related by D T V (p||q) ≤ � D KL (p||q)/2. Combining this with Jensen's inequality, we obtain E s∼d π [D T V (π � ||π)[s]] ≤ E s∼d π � � 1 2 D KL (π � ||π)[s] � ≤ � 1 2 E s∼d π [D KL (π � ||π)[s]] (7) From (  7 ) we immediately obtain the following. Corollary 3. In bounds (4), (  5 ), and (6), make the substitution E s∼d π [D T V (π � ||π)[s]] → � 1 2 E s∼d π [D KL (π � ||π)[s]]. The resulting bounds hold.
 Trust Region Methods Trust region algorithms for reinforcement learning  (Schulman et al., 2015;  have policy updates of the form π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] s.t. DKL (π||π k ) ≤ δ, (8) where DKL (π||π k ) = E s∼π k [D KL (π||π k )[s]], and δ > 0 is the step size. The set {π θ ∈ Π θ : DKL (π||π k ) ≤ δ} is called the trust region. The primary motivation for this update is that it is an approximation to optimizing the lower bound on policy performance given in (5), which would guarantee monotonic performance improvements. This is important for optimizing neural network policies, which are known to suffer from performance collapse after bad updates  (Duan et al., 2016) . Despite the approximation, trust region steps usually give monotonic improvements  (Schulman et al., 2015; Duan et al., 2016)  and have shown state-of-the-art performance in the deep RL setting  (Duan et al., 2016; Gu et al., 2017) , making the approach appealing for developing policy search methods for CMDPs. Until now, the particular choice of trust region for (8) was heuristically motivated; with (5) and Corollary 3, we are able to show that it is principled and comes with a worstcase performance degradation guarantee that depends on δ. Proposition 1 (Trust Region Update Performance). Suppose π k , π k+1 are related by (8), and that π k ∈ Π θ . A lower bound on the policy performance difference between π k and π k+1 is J(π k+1 ) − J(π k ) ≥ − √ 2δγ� π k+1 (1 − γ) 2 , ( 9 ) where � π k+1 = max s � � E a∼π k+1 [A π k (s, a)] � � . Proof. π k is a feasible point of (8) with objective value 0, so E s∼d π k ,a∼π k+1 [A π k (s, a)] ≥ 0. The rest follows by (5) and Corollary 3, noting that (8) bounds the average KLdivergence by δ. This result is useful for two reasons: 1) it is of independent interest, as it helps tighten the connection between theory and practice for deep RL, and 2) the choice to develop CPO as a trust region method means that CPO inherits this performance guarantee.
 Trust Region Optimization for Constrained MDPs Constrained policy optimization (CPO), which we present and justify in this section, is a policy search algorithm for CMDPs with updates that approximately solve (3) with a particular choice of D. First, we describe a policy search update for CMDPs that alleviates the issue of off-policy evaluation, and comes with guarantees of monotonic performance improvement and constraint satisfaction. Then, because the theoretically guaranteed update will take toosmall steps in practice, we propose CPO as a practical approximation based on trust region methods. By corollaries 1, 2, and 3, for appropriate coefficients α k , β i k the update π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] − α k � DKL (π||π k ) s.t. J Ci (π k ) + E s∼d π k a∼π � A π k Ci (s, a) 1 − γ � + β i k � DKL (π||π k ) ≤ d i is guaranteed to produce policies with monotonically nondecreasing returns that satisfy the original constraints. (Observe that the constraint here is on an upper bound for J Ci (π) by (  6 ).) The off-policy evaluation issue is alleviated, because both the objective and constraints involve expectations over state distributions d π k , which we presume to have samples from. Because the bounds are tight, the problem is always feasible (as long as π 0 is feasible). However, the penalties on policy divergence are quite steep for discount factors close to 1, so steps taken with this update might be small. Inspired by trust region methods, we propose CPO, which uses a trust region instead of penalties on policy divergence to enable larger step sizes: π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] s.t. J Ci (π k ) + 1 1 − γ E s∼d π k a∼π � A π k Ci (s, a) � ≤ d i ∀i DKL (π||π k ) ≤ δ. (10) Because this is a trust region method, it inherits the performance guarantee of Proposition 1. Furthermore, by corollaries 2 and 3, we have a performance guarantee for approximate satisfaction of constraints: Proposition 2 (CPO Update Worst-Case Constraint Violation). Suppose π k , π k+1 are related by (10), and that Π θ in (10) is any set of policies with π k ∈ Π θ . An upper bound on the C i -return of π k+1 is J Ci (π k+1 ) ≤ d i + √ 2δγ� π k+1 Ci (1 − γ) 2 , where � π k+1 Ci = max s � � E a∼π k+1 � A π k Ci (s, a) �� � .
 Practical Implementation In this section, we show how to implement an approximation to the update (10) that can be efficiently computed, even when optimizing policies with thousands of parameters. To address the issue of approximation and sampling errors that arise in practice, as well as the potential violations described by Proposition 2, we also propose to tighten the constraints by constraining upper bounds of the auxilliary costs, instead of the auxilliary costs themselves.
 Approximately Solving the CPO Update For policies with high-dimensional parameter spaces like neural networks, (  10 ) can be impractical to solve directly because of the computational cost. However, for small step sizes δ, the objective and cost constraints are well-approximated by linearizing around π k , and the KLdivergence constraint is well-approximated by second order expansion (at π k = π, the KL-divergence and its gradient are both zero). Denoting the gradient of the objective as g, the gradient of constraint i as b i , the Hessian of the KL-divergence as H, and defining c i . = J Ci (π k ) − d i , the approximation to (  10 ) is: θ k+1 = arg max θ g T (θ − θ k ) s.t. c i + b T i (θ − θ k ) ≤ 0 i = 1, ..., m 1 2 (θ − θ k ) T H(θ − θ k ) ≤ δ. (11) Because the Fisher information matrix (FIM) H is always positive semi-definite (and we will assume it to be positive-definite in what follows), this optimization problem is convex and, when feasible, can be solved efficiently using duality. (We reserve the case where it is not feasible  for   −1 2λ � g T H −1 g − 2r T ν + ν T Sν � + ν T c − λδ 2 , ( 12 ) where r . = g T H −1 B, S . = B T H −1 B . This is a convex program in m+1 variables; when the number of constraints is small by comparison to the dimension of θ, this is much easier to solve than (11). If λ * , ν * are a solution to the dual, the solution to the primal is θ * = θ k + 1 λ * H −1 (g − Bν * ) . ( 13 ) Our algorithm solves the dual for λ * , ν * and uses it to propose the policy update (13). For the special case where there is only one constraint, we give an analytical solution in the supplementary material (Theorem 2) which removes the need for an inner-loop optimization. Our experiments Algorithm 1 Constrained Policy Optimization Input: Initial policy π 0 ∈ Π θ tolerance α for k = 0, 1, 2, ... do Sample a set of trajectories D = {τ } ∼ π k = π(θ k ) Form sample estimates ĝ, b, Ĥ, ĉ with D if approximate CPO is feasible then Solve dual problem (12) for λ * k , ν * k Compute policy proposal θ * with (13) else Compute recovery policy proposal θ * with (14) end if Obtain θ k+1 by backtracking linesearch to enforce satisfaction of sample estimates of constraints in (10) end for have only a single constraint, and make use of the analytical solution. Because of approximation error, the proposed update may not satisfy the constraints in (10); a backtracking line search is used to ensure surrogate constraint satisfaction. Also, for high-dimensional policies, it is impractically expensive to invert the FIM. This poses a challenge for computing H −1 g and H −1 b i , which appear in the dual. Like  (Schulman et al., 2015) , we approximately compute them using the conjugate gradient method.
 Feasibility Due to approximation errors, CPO may take a bad step and produce an infeasible iterate π k . Sometimes (11) will still be feasible and CPO can automatically recover from its bad step, but for the infeasible case, a recovery method is necessary. In our experiments, where we only have one constraint, we recover by proposing an update to purely decrease the constraint value: θ * = θ k − � 2δ b T H −1 b H −1 b. (14) As before, this is followed by a line search. This approach is principled in that it uses the limiting search direction as the intersection of the trust region and the constraint region shrinks to zero. We give the pseudocode for our algorithm (for the single-constraint case) as Algorithm 1, and have made our code implementation available online. 1
 Tightening Constraints via Cost Shaping Because of the various approximations between (3) and our practical algorithm, it is important to build a factor of safety into the algorithm to minimize the chance of constraint violations. To this end, we choose to constrain upper bounds 1 https://github.com/jachiam/cpo on the original constraints, C + i , instead of the original constraints themselves. We do this by cost shaping: C + i (s, a, s � ) = C i (s, a, s � ) + Δ i (s, a, s � ), (15) where Δ i : S × A × S → R + correlates in some useful way with C i . In our experiments, where we have only one constraint, we partition states into safe states and unsafe states, and the agent suffers a safety cost of 1 for being in an unsafe state. We choose Δ to be the probability of entering an unsafe state within a fixed time horizon, according to a learned model that is updated at each iteration. This choice confers the additional benefit of smoothing out sparse constraints.
 Connections to Prior Work Our method has similar policy updates to primal-dual methods like those proposed by  Chow et al. (2015) , but crucially, we differ in computing the dual variables (the Lagrange multipliers for the constraints). In primal-dual optimization (PDO), dual variables are stateful and learned concurrently with the primal variables  (Boyd et al., 2003) . In a PDO algorithm for solving (3), dual variables would be updated according to ν k+1 = (ν k + α k (J C (π k ) − d)) + , (16) where α k is a learning rate. In this approach, intermediary policies are not guaranteed to satisfy constraints-only the policy at convergence is. By contrast, CPO computes new dual variables from scratch at each update to exactly enforce constraints.
 Experiments In our experiments, we aim to answer the following: • Does CPO succeed at enforcing behavioral constraints when training neural network policies with thousands of parameters? • How does CPO compare with a baseline that uses primal-dual optimization? Does CPO behave better with respect to constraints? • How much does it help to constrain a cost upper bound (15), instead of directly constraining the cost? • What benefits are conferred by using constraints instead of fixed penalties? We designed experiments that are easy to interpret and motivated by safety. We consider two tasks, and train multiple different agents (robots) for each task: Returns: Constraint values: (closer to the limit is better) • Circle: The agent is rewarded for running in a wide circle, but is constrained to stay within a safe region smaller than the radius of the target circle. • Gather: The agent is rewarded for collecting green apples, and constrained to avoid red bombs. For the Circle task, the exact geometry is illustrated in Figure  5  in the supplementary material. Note that there are no physical walls: the agent only interacts with boundaries through the constraint costs. The reward and constraint cost functions are described in supplementary material (Section 10.3.1). In each of these tasks, we have only one constraint; we refer to it as C and its upper bound from (15) as C + . We experiment with three different agents: a point-mass (S ⊆ R 9 , A ⊆ R 2 ), a quadruped robot (called an 'ant') (S ⊆ R 32 , A ⊆ R 8 ), and a simple humanoid (S ⊆ R 102 , A ⊆ R 10 ). We train all agent-task combinations except for Humanoid-Gather. For all experiments, we use neural network policies with two hidden layers of size (64, 32). Our experiments are implemented in rllab  (Duan et al., 2016) .
 Evaluating CPO and Comparison Analysis Learning curves for CPO and PDO are compiled in Figure  1 . Note that our constraint value graphs show C + return, instead of the C return (except for in Point-Gather, where we did not use cost shaping due to that environment's short time horizon), because this is what the algorithm actually constrains in these experiments. For our comparison, we implement PDO with (16) as the In Humanoid-Circle, the safe area is between the blue panels. update rule for the dual variables, using a constant learning rate α; details are available in supplementary material (Section 10.3.3). We emphasize that in order for the comparison to be fair, we give PDO every advantage that is given to CPO, including equivalent trust region policy updates. To benchmark the environments, we also include TRPO (trust region policy optimization)  (Schulman et al., 2015) , a stateof-the-art unconstrained reinforcement learning algorithm. The TRPO experiments show that optimal unconstrained behaviors for these environments are constraint-violating. We find that CPO is successful at approximately enforcing constraints in all environments. In the simpler environments (Point-Circle and Point-Gather), CPO tracks the constraint return almost exactly to the limit value. By contrast, although PDO usually converges to constraintsatisfying policies in the end, it is not consistently constraint-satisfying throughout training (as expected). For example, see the spike in constraint value that it experi-ences in Ant-Circle. Additionally, PDO is sensitive to the initialization of the dual variable. By default, we initialize ν 0 = 0, which exploits no prior knowledge about the environment and makes sense when the initial policies are feasible. However, it may seem appealing to set ν 0 high, which would make PDO more conservative with respect to the constraint; PDO could then decrease ν as necessary after the fact. In the Point environments, we experiment with ν 0 = 1000 and show that although this does assure constraint satisfaction, it also can substantially harm performance with respect to return. Furthermore, we argue that this is not adequate in general: after the dual variable decreases, the agent could learn a new behavior that increases the correct dual variable more quickly than PDO can attain it (as happens in Ant-Circle for PDO; observe that performance is approximately constraint-satisfying until the agent learns how to run at around iteration 350). We find that CPO generally outperforms PDO on enforcing constraints, without compromising performance with respect to return. CPO quickly stabilizes the constraint return around to the limit value, while PDO is not consistently able to enforce constraints all throughout training.
 Ablation on Cost Shaping In Figure  3 , we compare performance of CPO with and without cost shaping in the constraint. Our metric for comparison is the C return, the 'true' constraint. The cost shaping does help, almost completely accounting for CPO's inherent approximation errors. However, CPO is nearly constraint-satisfying even without cost shaping.
 Constraint vs. Fixed Penalty In Figure  4 , we compare CPO to a fixed penalty method, where policies are learned using TRPO with rewards R(s, a, s � ) − νC + (s, a, s � ) for ν ∈ {1, 5, 50}. We find that fixed penalty methods can be highly sensitive to the choice of penalty coefficient: in Ant-Circle, a penalty coefficient of 1 results in reward-maximizing policies that accumulate massive constraint costs, while a coefficient of 5 (less than an order of magnitude difference) results in cost-minimizing policies that never learn how to acquire any rewards. In contrast, CPO automatically picks penalty coefficients to attain the desired trade-off between reward and constraint cost.
 Discussion In this article, we showed that a particular optimization problem results in policy updates that are guaranteed to both improve return and satisfy constraints. This enabled the development of CPO, our policy search algorithm for CMDPs, which approximates the theoretically-guaranteed  algorithm in a principled way. We demonstrated that CPO can train neural network policies with thousands of parameters on high-dimensional constrained control tasks, simultaneously maximizing reward and approximately satisfying constraints. Our work represents a step towards applying reinforcement learning in the real world, where constraints on agent behavior are sometimes necessary for the sake of safety. the next subsection.) With B . = [b 1 , ..., b m ] and c . = [c 1 , ..., c m ] T , a dual to (11) can be expressed as max λ≥0 ν�0
 Figure 1 . 1 Figure1. Average performance for CPO, PDO, and TRPO over several seeds (5 in the Point environments, 10 in all others); the x-axis is training iteration. CPO drives the constraint function almost directly to the limit in all experiments, while PDO frequently suffers from over-or under-correction. TRPO is included to verify that optimal unconstrained behaviors are infeasible for the constrained problem.
 Figure 2 . 2 Figure 2. The Humanoid-Circle and Point-Gather environments.In Humanoid-Circle, the safe area is between the blue panels.
 Figure 3. Using cost shaping (CS) in the constraint while optimizing generally improves the agent's adherence to the true constraint on C return.
 Figure 4 . 4 Figure 4. Comparison between CPO and FPO (fixed penalty optimization) for various values of fixed penalty.