Abstract: Achieving the global benefits of artificial intelligence (AI) will require international cooperation on many areas of governance and ethical standards, while allowing for diverse cultural perspectives and priorities. There are many barriers to achieving this at present, including mistrust between cultures, and more practical challenges of coordinating across different locations. This paper focuses particularly on barriers to cooperation between Europe and North America on the one hand and East Asia on the other, as regions which currently have an outsized impact on the development of AI ethics and governance. We suggest that there is reason to be optimistic about achieving greater cross-cultural cooperation on AI ethics and governance. We argue that misunderstandings between cultures and regions play a more important role in undermining cross-cultural trust, relative to fundamental disagreements, than is often supposed. Even where fundamental differences exist, these may not necessarily prevent productive cross-cultural cooperation, for two reasons: (1) cooperation does not require achieving agreement on principles and standards for all areas of AI; and (2) it is sometimes possible to reach agreement on practical issues despite disagreement on more abstract values or principles. We believe that academia has a key role to play in promoting cross-cultural cooperation on AI ethics and governance, by building greater mutual understanding, and clarifying where different forms of agreement will be both necessary and possible. We make a number of recommendations for practical steps and initiatives, including translation and multilingual publication of key documents, researcher exchange programmes, and development of research agendas on cross-cultural topics.

Seán S Óhéigeartaigh
Jess Whittlestone
Yang Liu
Yi Zeng
Zhe Liu

Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance

Introduction Artificial intelligence has been identified as a key suite of technologies for many countries worldwide, in large part motivated by its general-purpose nature  (Brynjolfsson and McAfee 2014) . AI technologies, and machine learning techniques in particular, are being fruitfully applied to a vast range of domains, including language translation, scientific research, education, logistics, transport and many others. It is clear that AI will affect economies, societies and cultures profoundly at a national, international and global level. This has resulted in increasing attention being paid to both AI ethics: questions about how we should develop and deploy AI systems, given their potential impact on wellbeing and other deeply held values such as autonomy or dignity; and to AI governance: the more practical challenge of ensuring the ethical use of AI in society, be that through regulation, governance frameworks, or 'softer' approaches such as standards and ethical guidelines.  1  Cross-cultural cooperation will be essential for the success of these ethics and governance initiatives. By 'cross-cultural cooperation', we mean groups from different cultures and nations working together on ensuring that AI is developed, deployed, and governed in societally beneficial ways. In this paper, we focus in particular on cooperation extending across national boundaries. Examples include (but are not limited to) the following: AI researchers from different countries collaborating on projects to develop systems in safe and responsible ways; establishing networks to ensure that diverse global perspectives can feed equally into international discussions about the ethical issues raised by AI; and involving a range of global stakeholders in the development of practical principles, standards, and regulation. By encouraging crosscultural cooperation, we do not necessarily mean that all parts of the world should be subject to the same norms, standards, and regulation relating to AI, or that international agreement is always needed. Identifying which issues will need global standards or agreements, and where more cultural variation is needed, is itself a key challenge that will require cooperation to address. Cross-cultural cooperation is important for several reasons. First, cooperation will be essential if AI is to bring about broad benefits across societies globally, enabling advances in one part of the world to be shared with other countries, and ensuring that no part of society is neglected or disproportionately negatively impacted by AI. Second, cooperation enables researchers around the world to share expertise, resources, and best practices. This enables faster progress both on beneficial AI applications, and on managing the ethical and safety-critical issues that may arise. Third, in the absence of cooperation, there is a risk that competitive pressures between states or commercial ecosystems may lead to underinvestment in safe, ethical, and socially beneficial AI development  (Askell et al. 2019; Ying 2019) . Finally, international cooperation is also important for more practical reasons, to ensure that applications of AI that are set to cross-national and regional boundaries (such as those used in major search engines or autonomous vehicles) can interact successfully with a sufficient range of different regulatory environments and other technologies in different regions  (Cihon 2019) . Drawing on the insights of a group of leading scholars from East Asia and Europe, 2 we analyse current barriers to cross-cultural cooperation on AI ethics and governance, and how they might be overcome. We focus on cooperation between Europe and North America on the one hand and East Asia on the other. These regions are currently playing an outsized role in influencing the global conversation on AI ethics and governance  (Jobin et al. 2019) , and much has been written recently about competition and tensions between nations in these regions in the domains of AI development and governance, especially in the case of China and the USA. However, our discussion and recommendations have implications for broader international cooperation around AI, and we hope they will spur more attention to promoting cooperation across a wider range of regions. As AI systems become more capable and their applications more impactful and ubiquitous, the stakes will only get higher. Establishing cooperation over time may become more difficult, especially if cross-cultural misunderstandings and mistrust become entrenched in intellectual and public discussions. If this is the case, then the earlier a global culture of cooperation can be established, the better. Cultivating a shared understanding and deep cooperative relationships around guiding the impacts of AI should therefore be seen as an immediate and pressing challenge for global society. 2 The Role of North America, Europe, and East Asia in Shaping the Global AI Conversation North America, Europe, and East Asia in particular are investing heavily in both fundamental and applied AI research and development  (Benaich and Hogarth 2019; Haynes and Gbedemah 2019; Perrault et al. 2019) , supported by corporate and government investment. Various analyses have framed progress on the development and deployment of AI between the USA and China in particular through a competitive lens  (Simonite 2017; Allen and Husain 2017; Stewart 2017) , though this framing has received criticism both on normative and descriptive grounds (Cave and ÓhÉigeartaigh 2018). Scholars and policy communities in these regions are also taking deliberate and active steps to shape the development of ethical principles and governance recommendations for AI, both at a regional and global level. This is reflected in governmentlinked initiatives, such as the activities of the European Union's High-Level Expert Group on Artificial Intelligence, which has produced ethics guidelines and policy and 2 We convened a workshop on cross-cultural trust in July 2019 (https://www.eastwest.ai/), with representatives from UK universities (Cambridge, Bath) as well as from Chinese and Japanese universities and initiatives (Universities of Hong Kong, Peking, Fudan, Keio, and the Berggruen Institute China Center and the Chinese Academy of Sciences). These representatives have been heavily involved in AI ethics and governance conversations and collaborative projects across Europe, North America, and Asia. This paper draws on insights from this workshop, which focused particularly on the role of academia in building crosscultural trust in AI. We do not suggest that the regions represented are the only ones that matter in this conversation, nor that the members present can be considered to represent the diversity of views and expertise in the regions they are based in. However, we feel that this small step in establishing a network was of value, and generated useful insights and ground to build on. The workshop was held under Chatham House Rule. investment recommendations as its first two publications 3 ; the UK Government's commitments to 'work closely with international partners to build a common understanding of how to ensure the safe, ethical and innovative deployment of Artificial Intelligence'  (May 2018) ; and the Chinese Government's similar commitment to 'actively participate in global governance of AI, strengthen the study of major international common problems such as robot alienation and safety supervision, deepen international cooperation on AI laws and regulations, international rules' (China State Council 2017). North America, Europe, and East Asia are each also contributing disproportionately towards international AI standards work ongoing in fora such as the International Organization for Standardization (ISO), 4 the Institute of Electrical and Electronics Engineers (IEEE), 5 and the Organization for Economic Co-operation and Development (OECD).  6  The prominence of North America, Europe, and East Asia is further seen in the leadership and composition of multi-stakeholder and nongovernmental initiatives such as the Partnership on AI, 7 the Future Society, 8 the International Congress for the Governance of AI, 9 and the Global Partnership on AI  (Hudson 2019 ). 10 A large majority of the most prominent conferences on AI ethics and governance have taken place in these regions, including the US-based Beneficial AI conference series, 11 the Beijing Academy of AI Conference series, 12 the Beijing Forum, 13 the US-based Artificial Intelligence, Ethics, and Society conferences, 14 governance and ethics workshops attached to the leading machine learning conferences, and many more. This combination of: a. technological leadership in North America, Europe, and East Asia; b. the outsized role of these regions in shaping the global ethics and governance conversation; and c. the underlying tension introduced by progress being framed through a competitive lens, and a perception of disagreements on fundamental ethical and governance issues leads us to focus on the barriers that exist to productive intellectual exchange and cooperation between these regions and cultures in particular. A full analysis of cross-cultural cooperation on AI ethics and governance, which is outside the scope of this paper, must consider the roles of all nations and cultures, given that the domain of impact of AI technologies is truly global. It will be particularly important for further work to address the inequalities in power and influence that are emerging between technology-leading nations and those to which these technologies are being exported  (Lee 2017) , and the responsibility of technology-leading nations to include and empower those nations in global governance and ethics conversations. 3 Barriers to Cross-cultural Cooperation on AI Despite the emergence of several international alliances in AI ethics and governance, many barriers remain to achieving real cross-cultural cooperation on the norms, principles, and governance frameworks that should guide how AI is developed and used. Mistrust between different regions and cultures is one of the biggest barriers to international cooperation in AI ethics and governance. At present, there is a particular environment of mistrust between scholars, technologists, and policymakers in the USA and China.  15  This culture of mistrust is underpinned by both: a. a history of political tensions between these two powerful regions that has increased significantly in recent years, and is currently contributing to the competitive framing of AI development as a 'race' between 'Eastern' and 'Western' nations.  16  and b. the divergent philosophical traditions upon which these regions are founded, leading to a perception of significant and irresolvable value differences between 'Western' and 'Eastern' cultures on key issues such as data privacy  (Larson 2018; Horowitz et al. 2018; Houser 2018) . A range of recent technological and political developments may also be contributing to this mistrust, including concerns about the public and political influence of technology giants in the USA  (Ochigame 2019) ; perceptions of and reactions to the Chinese Social Credit score system  (Chorzempa et al. 2018; Song 2019) ; and concerns about contentious uses of AI technologies, with notably controversial examples including the use of AI in immigration control  (Whittaker et al. 2018) , criminal risk assessment in the USA  (Campolo et al. 2017) , and in tracking communities such as the Uighur Muslim minority in China  (Mozur 2019) . Adversarial rhetoric from political and defence leaders in the USA also contributes to this tension. Recent examples reported in the media include stating intentions to 'be the threat' of AI 17 ; comments focused on the 'otherness' of China as an adversary, 18 amid broader concerns regarding Chinese technological progress as a threat to US global leadership  (Jun 2018) . A continued exacerbation of this culture of mistrust could severely undermine possibilities for global cooperation on AI development and governance. In addition, it is unclear how far existing cross-cultural collaborations and alliances can go to shape the behaviour of actors that are as globally dominant as the USA, China, and the large multinational corporations based in these countries. Even if AI ethics frameworks can be agreed on in principle by multi-stakeholder groups, for example, it will be far from straightforward to implement them in practice to constrain the behaviour of those with disproportionate power to shape AI development and governance. Another challenge for effective cooperation is balancing the need for global cooperation with the need for culturally and geographically sensitive differentiated approaches  (Hagerty and Rubinov 2019) . It is crucial we avoid a situation where one or two nations simply try to impose their values on the rest of the world  (Acharya 2019) . In certain specific domains, for example where AI is being used to support the delivery of healthcare, different cultures may perceive tradeoffs very differently  (Feldman et al. 1999) , and it may be not just possible but necessary to implement region-specific standards and governance. AI systems will also have different impacts as they are deployed in different cultural regions, which may also require different governance approaches  (Hagerty and Rubinov 2019) . For some aspects of AI development and governance, however, cooperation will be much more crucial. For example, some potential uses of AI technologies in military contexts, such as in automated targeting and attack, could impinge upon human rights and international humanitarian law  (Asaro 2012) . Another concern is that by automating aspects of information gathering, decision-making, and response in military arenas, the potential for unwanted escalation in conflict situations may increase due to events occurring and requiring responses faster than is compatible with effective human judgement and oversight  (Altmann 2019) . In both these cases, there may be individual military advantages to nations pursuing the technology; but in the absence of international agreements and standards, the overall effect may be destabilizing.  19  International agreement will also be of particular importance for all cases in which AI technologies are developed in one region, but used or deployed in a different region. A key challenge for cross-cultural cooperation will therefore be to identify the areas where international agreement is most important, and distinguish these from areas where it is more appropriate to respect a plurality of approaches. There are also more practical barriers to cooperation between nations: language barriers, lack of physical proximity, and immigration restrictions put limits on the ability of different cultures and research communities to communicate and collaborate. Furthermore, despite science being a global enterprise, the language of scientific publication remains predominantly English.
 Overcoming These Barriers to Cooperation While cross-cultural cooperation in AI ethics and governance will be genuinely challenging, we suggest that there are steps that can be taken today to make progress, without first needing to tackle the greater problems of finding consensus between cultures on all fundamental ethical and philosophical issues, or resolving decades of political tension between nations.
 Building Greater Mutual Understanding, Including Around Disagreements Mistrust between nations is a serious concern for the future of AI ethics and governance. However, we suggest that this mistrust is at least partly fuelled by misunderstandings and misperceptions, and that a first step towards building greater crosscultural trust must therefore be to identify and correct important misperceptions, and enhance greater mutual understanding between cultures and nations. It would be easy to assume that the main barrier to building trust between East and West around AI is that these regions of the world have very different fundamental values, leading them to different-perhaps conflicting-views of how AI should be developed, used, and governed from an ethical perspective. While value differences between cultures certainly exist, claims about how those differences manifest often depend on unexamined concepts and entrenched assumptions, and lack empirical evidence  (Whittlestone et al. 2019) . The idea that 'Eastern' and 'Western' ethical traditions are fundamentally in conflict also oversimplifies the relationship between the two. There are many different philosophical traditions that might be referred to under either heading: there are, for example, many important differences between relevant philosophical perspectives across China, Japan, and Korea  (Gal 2019) , and the values and perspectives of 'Western' philosophy have changed a great deal over time  (Russell 1945) . More broadly, ethical and cultural values in both regions are in constant evolution, as captured by projects such as the World Values Survey 20 and the Asian Barometer.  21  Differences in ethical and cultural traditions and norms across regions are often assumed to underpin contrasting governance approaches. For example, privacy is often seen as an issue for which significant value differences exist between East and West, leading to a perception that laxer regulation and controls exist on data privacy in China compared with the USA and Europe. However, such claims are often made in very broad terms, without substantive evidence or analysis of how significant these differences are or how they manifest in practice  (Ess 2005; Lü Yao-Huai 2005) . This leads to misunderstandings in both directions. First, there are significant differences between the USA and Europe on both conceptions of privacy  (Szeghalmi 2015)  and regulations that relate to privacy  (McCallister et al. 2018) . These are often missed in Chinese perceptions of Western societies, which tend to focus just on the USA.  22  Second, Western perceptions of data privacy in China may be outdated: Lü Yao-Huai (2005) highlighted as early as 2005 that the relevant literature on information ethics was much younger in China than in the USA, but was evolving quickly and in a manner significantly informed by Western scholarship. A range of papers and reports from Chinese scholars and policymakers have highlighted the importance of data privacy in AI ethics and governance (Beijing Academy of Artificial Intelligence 2019; Ying 2019;  Zeng et al. 2018; Ding 2018b ). Principles around protecting individuals' data privacy are also beginning to be borne out in regulatory action in China; over 100 apps have been banned by the government for user data privacy infringements, with dozens more being required to make changes relating to data collection and storage.  23  This is not to suggest that there are not meaningful differences in values, norms, and regulations relating to data privacy between these countries, but that such differences have often been oversimplified and are not well understood. Another example of differing perceptions are those surrounding China's social credit score (SCS) system. The SCS has been discussed with great concern in Western media, policy circles, and scholarship, and presented as an example of Orwellian social control by the Chinese government  (Botsman 2017; Pence 2018) , representative of a culture and government with profoundly different values to the West  (Clover 2016 ). However, both Chinese and Western sources have argued that there are significant misunderstandings surrounding the SCS. Multiple scholars have pointed out that the SCS is not designed to be a single unified platform that rates all 1.4 billion Chinese citizens (as is often supposed), but rather a web of individual platforms with latitude for different interpretations, with social credit scores mostly given by financial institutions (as opposed to a big data-driven comprehensive rating)  (Mistreanu 2019; Sithigh and Siems 2019) .  Song (2019)  notes that many of the measures in the SCS are designed to tackle issues such as fraud and corruption in local government.  Chorzempa et al. (2018)  also highlight that 'many of the key components of social credit, from blacklists to widespread surveillance… already exist in democracies like the United States.' China's SCS is likely to evolve significantly over time, and there are likely to be genuine reasons for concern both in terms of present and future implementation. However, a much clearer cross-cultural understanding of how the SCS works, is being used, and is impacting Chinese citizens, would allow dialogue on relevant ethics and governance issues to progress more constructively. Given the lack of shared knowledge and discourse that has existed historically between regions such as the USA, Europe, and China, it is not surprising that many  22  This was a misperception noted by multiple Chinese scholars at the aforementioned July workshop.  23  In November 2019, the Chinese Ministry of Public Security banned 100 apps that failed to meet standards on individuals' data privacy; the body has investigated 683 apps in 2019 (National Cyber Security Advisory Centre 2019). In addition, in December 2019, the Chinese Ministry of Industry and Information Technology released a list of 41 apps that would have to make changes to comply with data regulations by the end of 2019 (Ministry of Industry and Information Technology of the People's Republic of China 2019). In July 2018, China's Shandong Province brought a major case relating to infringement of personal information against 11 companies  (Ding 2018c).  misperceptions exist between them. We should therefore be wary of jumping too quickly to assume intractable and fundamental disagreements. Misunderstandings clearly exist in both directions: analyses of public opinion survey data suggest, for example, that both American and Chinese populations hold various misperceptions about the other nation's traits and characteristics (Johnston and Shen 2015). As mentioned above, in China, the diversity of Western societies is also often oversimplified down to a single pattern of American life. At the same time, the USA and Europe have historically struggled to understand China (Chen and Hu 2019), evidenced for example by repeated failures to predict China's liberalization (or lack thereof) or periods of economic growth (The Economist 2018; Cowen 2019; Liu 2019). Language barriers present a particular difficulty for Western nations in gleaning what is happening in China in terms of AI development, ethics, and governance  (Zhang 2017; ). As Andrew Ng points out in a 2017 interview in the Atlantic: 'The language issue creates a kind of asymmetry: Chinese researchers usually speak English so they have the benefit of access to all the work disseminated in English. The Englishspeaking community, on the other hand, is much less likely to have access to work within the Chinese AI community'  (Zhang 2017) . For example, Tencent released a book on AI strategy  (Tencent Research Institute et al. 2017)  which includes deep analysis of ethics, governance, and societal impacts, but has received relatively little English-language coverage (Ding 2018a). Even on empirical matters such as level of Chinese public investment in AI research and development, widely reported figures in the USA may be inaccurate by an order of magnitude  (Acharya and Arnold 2019) . The recently published Beijing AI Principles (Beijing Academy of Artificial Intelligence 2019) and similar principles developed around the world  (Cowls and Floridi 2018)  in fact show substantial overlap on key challenges  (Zeng et al. 2018; Jobin et al. 2019) . The Beijing Principles make clear reference to the key concepts and values which have been prominent in other documents, including that AI should 'benefit all humankind'; respect 'human privacy, dignity, freedom, autonomy and rights'; and be 'as fair as possible, reducing possible discrimination and biases, improving its transparency, explainability and predictability.' In addition, both the Beijing AI Principles, and the National Governance Principles of the New AI, China, call for openness and collaboration, with the latter encouraging 'cooperation across disciplines, domains, regions, and borders'  (Laskai and Webster 2019) . However, nations with different cultures may interpret and prioritize the same principles differently in practice  (Whittlestone et al. 2019) , which may be a further source of misunderstanding. We cannot simply assume, for example, that 'Western' cultures value privacy more highly than 'Eastern' ones; instead, we need a more nuanced understanding of how privacy may be prioritized differently when it comes into conflict with other important values, such as security  (Capurro 2005) . Similarly, although it is important to recognize that many cultures value autonomy, it is equally important to understand the different connotations and philosophical assumptions underpinning this value in different contexts  (Yunping 2002) . Given the rich history of misunderstanding between nations, to build greater crosscultural cooperation, we should start by focusing on identifying those misperceptions most relevant to AI ethics and building greater mutual understanding of where more substantive disagreements exist and are likely to impact governance approaches. In doing so, it is worth distinguishing explicitly between disagreements pertaining to ethics as opposed to governance issues, since it may sometimes be possible for groups to agree on the same governance principles despite justifying them with different ethical assumptions, as we will discuss later. It may also be helpful to distinguish between misunderstandings that pertain directly to AI (such as misperceptions of other countries' investment in technology, or misinterpretation of data protection laws) and those that pertain to broader societal, political, or philosophical matters that are more indirectly relevant to AI, as they may require different approaches to resolve. Acknowledging the role of misunderstandings does not, of course, imply that all matters of intercultural tension in AI ethics and governance are fundamentally based on misunderstandings. Deep and fundamental disagreements across regions will remain on a range of issues, including those relating to the relationship between the individual, society, and the state; the level and nature of integration between civil, private, and military sectors; and various specific matters of social policy. However, focusing initially on reducing misunderstandings will aid in establishing more clearly where these fundamental differences exist, while at the same time identifying contexts in which sufficient agreement exists for fruitful cooperation. Doing so is a crucial first step towards addressing the broader challenges of cross-cultural cooperation on AI ethics and governance.
 Finding Ways to Cooperate Despite Disagreements Even where important differences of view on AI ethics, governance, and broader societal issues exist, forms of agreement and cooperation can still be possible. As mentioned earlier, a key outstanding challenge for AI ethics and governance is identifying those areas where cross-cultural agreement on norms, standards, or regulation is crucial, and where different interpretations and approaches are acceptable or even desirable. This is precisely the kind of challenge which itself requires crosscultural cooperation: the delineations must be informed by diverse cultural perspectives on the impacts of AI in different contexts, and the needs and desires of different populations. Indeed, this approach is reflected in the National Governance Principles of the New AI, China, which includes the recommendation to 'Launch international dialogue and cooperation; with full respect for each country's principles and practices for AI governance, promote the formation of a broad consensus on an international AI governance framework, standards, and norms'  (Laskai and Webster 2019) . Regional and cultural differences on the abstract level of ethical assumptions and high-level principles are also not necessarily a barrier to agreement on more concrete norms and governance. If it were impossible to reach any practical agreement without consensus on fundamental ethical issues, many important international agreementssuch as the Nuclear Weapons Ban Treaty-would not have been possible. The notion of an 'incompletely theorized agreement' in legal scholarship  (Sunstein 1995) , describes how it is often possible for people who disagree on fundamental or abstract matters to nonetheless agree on specific cases-and that this is central to the functioning of law as well as of a pluralistic society more broadly. Several authors in the literature on intercultural information ethics have promoted the related idea of aiming to arrive at an 'overlapping consensus'  (Rawls 1993) , where different groups and cultures may have different reasons for supporting the same norms and practical guidelines  (Taylor 1996; Søraker 2006; Hongladarom 2016) . For example,  Taylor (1996)  discusses how we have managed to ground shared norms of human rights in different cultural traditions. While Western philosophies differ substantially from others such as Buddhism in how much importance they give to the human agent and its unique place in the cosmos, both seem to end up grounding the same norms of human rights.  Wong (2009)  criticizes this idea that intercultural information ethics can arrive at shared norms with different justifications, suggesting that this risks making norms too 'thin', devoid of all normative content.  Søraker (2006)  acknowledges a similar objection to this 'pragmatic' approach to information ethics: that it may result in agreements that are fragile due to not being sufficiently grounded in substantive normative content. However, in line with Søraker's own response to these objections, we believe that the aim of 'overlapping consensus' should be to arrive at shared norms and practical guidelines which are in fact more robust by virtue of being endorsed and justified from a range of different philosophical or normative perspectives. This should be distinguished from a situation where one culture uses pragmatic arguments to attempt to force their own values upon others, or where several cultures reach agreement but for reasons with little normative content, which, we agree with Wong, would be concerning. Taylor's example of human rights being supported by multiple philosophical perspectives appears to demonstrate the plausibility of this kind of wellsubstantiated overlapping consensus. Indeed, we suggest that finding areas of overlapping consensus on norms and practical guidelines may be much more important for ensuring the benefits of AI than aiming for global consensus on a shared set of fundamental values-an aim which underpins many recent proposals.  24  Consensus on high-level ethical principles does not necessarily mean they are well-justified  (Benjamin 1995) , and the best way to arrive at more robustly justified norms, standards, and regulation for AI will be to find those that can be supported by a plurality of different value systems.
 Putting Principles into Practice Even where it is possible to improve mutual understanding and arrive at shared governance approaches in theory, some might object that it will still be difficult to influence the development and use of AI in practice, especially since to do so requires influencing the behaviour of powerful states and companies that have little incentive to cooperate. Although a full discussion of the complex power dynamics between states, companies, and other actors on issues relevant to AI ethics and governance is beyond the scope of this paper (and worthy of further research), we briefly explain why we do not think this barrier undermines our proposals. While challenging, historical precedent suggests that it is possible for public and academic alliances to influence the behaviour of powerful actors on issues of global importance. There is evidence to suggest that broad, cross-cultural 'epistemic communities'-i.e. networks of experts in a particular domain-can be particularly effective at supporting international policy coordination (Haas 1992). For example, a community of control experts helped shape coopbetween the USA and Russia during the Cold War by creating an internationally shared understanding of the problem of nuclear arms control  (Adler 1992) , and the ecological epistemic community managed to successfully coordinate national policies to protect the stratospheric ozone layer  (Haas 1992) . The commitments of large companies and even nations around AI have already been influenced by combinations of employee activism, international academic research, and campaigning. A notable example is in the application of AI in military contexts. Concerns over the use of AI in warfare have been the subject of high-profile campaigns by experts across academia and civil society internationally, such as those involved in the International Committee for Robot Arms Control (ICRAC) and the Campaign to stop Killer Robots. These campaigns played a leading role in establishing discussion on lethal autonomous weapons (LAWs) at the United Nations Convention on Certain Conventional Weapons (CCW; the body that hosted negotiations over the banning of cluster munitions, blinding laser weapons, and landmines)  (Belfield 2020) . Ninety countries have put forward statements on LAWs, with most doing so at the CCW; 28 countries support a ban (Campaign to Stop Killer Robots 2018).  25  In 2018, over 4000 Google employees signed a letter protesting Google's involvement in the Pentagon's Project Maven, a military project exploring the use of AI in footage analysis  (Shane and Wakabayashi 2018) , and a number resigned  (Conger 2018) . Several other campaigning groups, comprised of scholars and researchers from the USA, Europe, Japan, China, Korea and elsewhere, released public articles and letters supporting the concerns of the Google practitioners (ICRAC 2018).  26  Google subsequently announced it would not renew its contract on Project Maven, and would not bid on a $10 billion Department of Defence cloud computing contract  (Belfield 2020) . More broadly, international academic and civil society input has played a significant role in shaping principles that are likely to form the basis for binding regulation in years to come. For example, the European Commission's white paper On Artificial Intelligence -A European Approach to Excellence and Trust (European Commission 2020) lays out 'policy options for a future EU regulatory framework that would determine the types of legal requirements that would apply to relevant actors, with a particular focus on high-risk applications'(European Commission 2020b).This document was strongly influenced by the work of the European Union's High-Level Expert group on Artificial Intelligence, comprising 52 European experts from across academia, industry, and civil society.  27  Similarly, the US Department of Defence has formally adopted ethical principles on AI (US Department of Defense 2020), after 15 months of consultation with US-based academic, industry, and government stakeholders, and has hired staff to implement these principles  (Barnett 2020) . While in both cases the groups consulted were region-specific, the degree of alignment and overlap between principles  25  China supports a ban on use of fully autonomous weapons on the battlefield, but not their production and development. The USA, Russia, UK, Israel, and France oppose a ban.  (Kania 2018 )  26  An open letter was released by the International Committee for Robot Arms control and signed by over 1000 scholars and researchers, and members of the Campaign to Stop Killer Robots wrote public articles and letters to company leaders supporting the concerns of the Google petitioners: https://www.stopkillerrobots. org/2019/01/rise-of-the-tech-workers/ 27 Information on process and composition available at: https://ec.europa.eu/digital-single-market/en/highlevel-expert-group-artificial-intelligence developed in different regions suggests the insights and recommendations are tially informed by interaction with broader epistemic communities from regions. This suggests that insights gained from cross-cultural cooperation and consensus can meaningfully feed into regulatory frameworks at a regional and national level.
 Recommendations Academia has an important role to play in supporting cross-cultural cooperation on AI ethics and governance: both through research into where and what kinds of cooperation are most needed, and by establishing initiatives to overcome more practical barriers to cooperation. Our discussion in this paper raises many questions that will require diverse academic expertise to answer, including questions about what important misperceptions most hinder cooperation across regions; where international agreement is needed on AI ethics and governance; and how agreement might be reached on specific governance standards despite differences on ethical issues. Academic communities are also particularly well-suited to building greater mutual understanding between regions and cultures in practice, due to their tradition of free-flowing, international, and intercultural exchange of ideas. Academics can have open conversations with international colleagues in a way that is often challenging for those working in industry or government, and two academics from different parts of the world can productively collaborate even if each has strong criticisms of the other nation's government and/or companies. The following recommendations indicate a number of specific steps that academic centres, research institutions, and individual researchers can take to promote crosscultural understanding and cooperation on AI ethics and governance. Some excellent work is already ongoing in each of these areas. However, we believe that the pace at which AI is being deployed in new domains and regions calls for a greater focus within the academic community on building cross-cultural bridges and incorporating crosscultural expertise within a wider range of ethics and governance research projects. Develop AI Ethics and Governance Research Agendas Requiring Cross-cultural Cooperation Greater cross-cultural collaboration on research projects will play a crucial role in building an international research community that can support international policy cooperation (Haas 1992). An example of a research project that might be well-suited to such collaboration is to conduct comparative foresight exercises exploring differences in how both positive visions and prominent concerns about AI's impact on society vary across cultures. This could help with developing a more global vision for what we hope to both achieve and avoid in AI development, which could guide more practical discussions around ethics and governance frameworks. There may be particularly valuable opportunities for consensus generation around avoidance of particular negative outcomes; safety and security are fundamental to human cultures worldwide, and so developing agreements to avoid threats to these may be an easier starting point. However, the authors feel that it is important not to neglect positive visions, as the opportunity for scholars across cultures to co-create shared positive futures may represent an excellent way to delve into nuances within shared values. It would also be particularly valuable to the ongoing and expected impact of AI on developing countries, in collaboration with experts these countries. Such research should aim ensure that decisions to deploy technology in developing nations are made with the guidance of local expertise in such a way as to empower local communities  (Hagerty and Rubinov 2019) . On a more practical level, international research groups could productively work together to develop frameworks for international sharing of research, expertise, and datasets on AI safety, security, and avoidance of societal harm. Collaboration between researchers from multiple regions and cultures will also be essential to further research on the topic of cross-cultural cooperation itself. Our discussion in this paper, especially in section 4, has pointed to many research areas in need of further exploration, including the following: & Exploring, identifying, and challenging perceived cross-cultural differences in values, assumptions, and priorities relevant to AI ethics and governance, on multiple different levels, including 28 the following: -Analysing similarities and differences in technology ethics across different philosophical traditions, and exploring how these may affect AI development, deployment, impacts, and governance in practice; -Exploring the empirical evidence behind claims of key value differences between cultures. For example, a project might identify and explore perceived value differences between Eastern and Western cultures relevant to AI governance, such as those relating to data privacy, the role of the state vs. the individual, and attitudes towards technological progress; -Understanding regional differences in practical priorities and constraints relating to the use of AI in society, and the implications of these differences for AI research and development. & Further analysis to identify aspects of AI governance where global agreement is needed, and differentiating these from areas in which cross-cultural variation is either acceptable or desirable; & Cross-cultural contribution to the development of international and global AI standards in key domains for which these are needed; exploration of flexible governance models that allow for a combination of global standards and regional adaptability where appropriate; & Exploring models and approaches for reaching agreement on concrete cases, decisions, or governance standards despite disagreement on more fundamental or abstract ethical issues, and identifying cases of this being done successfully in other domains that can be translated to AI ethics and governance. Some excellent work is already ongoing in each of these areas. However, we believe that the pace at which AI is being deployed in new domains and regions calls for a greater focus within the academic community on bridges, and incorporating cross-cultural expertise, within a wider range of ethics and governance research projects.  29  Translate Key and Reports Language is a major practical barrier greater cross-cultural understanding around AI development, governance and ethics, as it has been in many other areas of science  (Amano et al. 2016) . It would therefore be extremely valuable for the burgeoning literature on AI ethics and governance, as well as the literature on AI research, to be available in multiple languages. While many leading Asian scholars in AI are fluent in English, many are not; and the fraction of Western researchers fluent in Mandarin or Japanese is far lower. Furthermore, some sources of the misunderstandings we have discussed may link to the ways in which key documents from one region are understood and presented in other regions. In Western media, China's 2017 New Generation Artificial Intelligence Development Plan has been presented as promoting the aim of global dominance in AI economically and strategically  (Knight 2017; Demchak 2019) . However, from the Chinese perspective, national AI development goals appear to be primarily motivated by the needs of the Chinese economy and society  (China State Council 2017) , rather than necessarily international competitive superiority  (Ying 2019 ). It appears that some translations may have led to misinterpretations of key terms and points. For example, the original Chinese text referred to China becoming 'a primary AI innovation center of the world by 2030'  (Ying 2019) .  30  However, some English translations of the report translated this phrase as China becoming 'the world's primary AI innovation center' (e.g.  Webster et al. 2017) . This was then interpreted and presented by Eric Schmidt, former executive chairman of Google parent Alphabet, as 'By 2030 they will dominate the industries of AI. Just stop for a sec. The [Chinese] government said that.'  (Shead 2017) . While this evolution of wording is not substantial in one sense, it carries important connotations; the language in the original context carries much softer connotations of leadership and progress, as opposed to global dominance. Having multiple high-quality translations of the most important documents would allow scholars to explore nuances of language and context that may be lost in the reporting of these documents. Producing high-quality versions of papers and reports in multiple languages also conveys respect and an appetite to engage cross-culturally, which is likely to encourage cooperation. High-quality translation of academic and policy materials is a challenging and time-consuming task that we would encourage being supported and celebrated more strongly. There is a growing body of work to be celebrated in this vein; for example, Jeff Ding's translation of a range of key Chinese AI documents , the work of Intellisia in China on international relations, technology, and other topics, which publishes in 5 languages (http://www.intellisia.org/); Brian Tse's translation to Chinese of documents including OpenAI's Charter 31 ; and New America's translation of the Chinese Ministry of Industry and Information Technology's Three Year Action Plan  (Triolo et al. 2018) .
 Alternate Continents for Major AI Research Conferences Governance Conferences To encourage more global participation in AI development, ethics, and governance, we recommend that many of the leading conferences fora on these topics alternate between multiple continents. This has several advantages. It reduces cost and time commitment for scholars from parts of the world in which these conferences do not frequently take place to participate. It avoids restrictive visa limitations differentially affecting certain parts of the global research community. It encourages the involvement of local organizers, who can play an effective role in engaging local research communities who might not consider travelling far overseas for an event. It also encourages organizers to run events multilingually rather than monolingually. Again, there are encouraging steps. Of AI research conferences, IJCAI took place in Macau in 2019 and Beijing in 2013, the first two times the conference had been held in China (although it has been held in Japan twice). ICML took place in Beijing in 2014, and will be in Seoul in 2021, and ICLR 2020 will be held in Ethiopia, making it the first of the top tier major machine learning conferences to be held in Africa. There are fewer established conferences explicitly focused on AI ethics and governance since the field's growth is relatively recent, but it may be particularly important for these conferences to ensure a global presence by alternating the continent on which they are held if possible. AI Ethics and Society, for example, is currently held in the USA due to ties to AAAI; the importance of building an international academic community around these issues may justify finding some way to change this. There is a burgeoning set of AI ethics and governance conferences in China, including the Beijing Academy of AI Conference series. There are also several existing conferences which cover topics relevant to AI ethics and governance (even if not so explicitly centred around them), which do enable more international participation, such as the World Summit on the Information Society Forum (held most years in Geneva), the Internet Governance Forum, and RightsCon (which have both been held in a range of locations historically including in South America, India, and Africa, though neither in East Asia).  32  Establish Joint and/or Exchange Programmes for PhD Students and Postdocs Encouraging cross-cultural collaboration between researchers from different cultures early on in their careers will help support greater cooperation and mutual understanding as research advances. Many international fellowships and exchange programmes exist, especially between the USA and China (e.g. the Zhi-Xing China Fellowship and the Schwarzman Scholars programme) as well as between the UK and Singapore (King's College London offers a joint PhD programme in Philosophy or English with the National University of Singapore). To our knowledge, no such initiatives exist explicitly focused on AI; the only initiative focused on AI ethics and governance that we are currently aware of is the international fellowship programme recently established by the Berggruen China Institute  (Bauch 2019) .  33  Establishing more such programmes could be valuable for the future of international cooperation around AI, and there are many existing models from which to build and More broadly, the authors endorse the Partnership on AI's recommendations to governments on establishing visa pathways, simplifying and expediting processes, and ensuring just standards to support international exchange and collaboration of AI/ ML multidisciplinary experts. We emphasize that these recommendations include experts working or seeking to work on AI ethics and governance (which sometimes fall outside of what is classed as 'skilled technology work') (PAI Staff 2019).
 Limitations and Future Directions We believe that academia has an important role to play in supporting cross-cultural cooperation in AI ethics and governance: that it is possible to establish effective communities of mutual understanding and cooperation without needing to resolve all fundamental value differences, and that reducing misunderstandings and misperceptions between cultures may be of particular importance. However, we recognize that the suggestions in this paper cannot go all the way to overcoming the many barriers to cross-cultural cooperation, and that much more work needs to be done to ensure AI will be globally beneficial. We briefly highlight two broad avenues of further research in support of this goal: More Detailed Analysis of Barriers to Cross-cultural Cooperation, Especially Those Relating to Power Dynamics and Political Tensions While analysis of historical successes suggest that it is possible for cross-cultural initiatives around AI ethics and governance to considerably shape how norms, standards, and regulation evolve in practice, there are still many barriers to implementation and enforcement that we were unable to consider in this analysis. Further research into when and how attempts to influence globally relevant norms and regulation have been successful in the past would be of considerable value. We acknowledged earlier in this paper that various issues related to power relations and political tensions likely pose significant barriers to cross-cultural cooperation, beyond problems of value differences and misunderstandings between cultures. More research on how these issues present barriers to cross-cultural cooperation in AI ethics and governance would therefore be particularly valuable, helping us to understand the limits of academic initiatives in promoting cooperation, and in what ways these approaches need to be embedded within a broader analysis of power and political dynamics. Considering the Unique Challenges of Cross-cultural Cooperation Around More Powerful Future AI Systems Future advances in AI, which some scholars have theorized could have impacts as transformative as the industrial or agricultural revolutions  (Karnofsky 2016; Zhang and Dafoe 2019) , may raise new challenges for global cooperation of a greater scale than we already face. Without careful global stewardship, such advances could lead to unprecedented inequalities in wealth and power between technology-leading and lagging nations. Others have gone further, theorizing about the possibility of developing systems exhibiting superintelligence (i.e. greater-than-human general intelligence; Bostrom 2014). Such systems, due to tremendous capability, might pose catastrophic risks human civilisation if without careful forethought and attention to safety. It has been proposed that a key for avoiding catastrophic outcomes will be value alignment-designing systems that are aligned with humanity's values  (Russell 2019) . This would greatly increase the importance and urgency of reaching global consensus on shared values and principles, as well as finding ways to design systems to respect values that are not shared. Expert views differ widely on how far in the future such advances might lie, with most predicting decades. However, developing the collaborations and agreements necessary for an effective and coordinated response may also require decades of work. This suggests that cooperative initiatives today must address not just the ethics and governance challenges of current AI systems, but should also lay the groundwork for anticipating and engaging with future challenges.
 Conclusion The full benefits of AI cannot be realized across global societies without a deep level of cooperation-across domains, disciplines, nations, and cultures. The current unease and mistrust between the USA and Europe on the one hand, and China on the other hand, places a particular strain on this. Misunderstandings may play an important role in fuelling this mistrust, and differences in broader societal and political priorities frequently appear to be overemphasized or misunderstood. At the same time, it would be naive to assume that all major ethical principles relating to AI can be shared in full between these regions, and can be enshrined in rules and standards. Even if this were the case, it would not be desirable for these regions to be overly dominant in shaping the ethics and governance of AI globally; all global communities to be affected by AI must be included and empowered. However, efforts to achieve greater understanding between these 'AI superpowers' may help in two ways: Firstly, by reducing key tensions within the global AI governance sphere. Secondly, by providing lessons that can contribute to ethics and governance frameworks capable of supporting a greater diversity of values while allowing consensus to be achieved where needed. For a well-functioning system of global cooperation in AI, the challenge will be to develop models that combine both principles and standards shaped and supported by global consensus, and the variation that allows research and policy communities to best serve the needs of their societies. On a more practical level, the international AI research community, and AI ethics and governance communities, must think carefully about how their own activities can support global cooperation, and a better understanding of different societal perspectives and needs across regions. Greater cross-cultural research collaboration and exchange, conferences taking place in different regions, and more publication across languages can lower the barriers to cooperation and to understanding different perspectives and shared goals. With political winds increasingly favouring isolationism, it has never been more important for the research community to work across national and cultural divides towards globally beneficial AI. 			 This perception of mistrust was highlighted throughout the July workshop. Several participants reported being aware of or present at workshops focused on geopolitical implications of Chinese AI progress in which Chinese participants were excluded and events at which 'what should be done about China' (from a Western perspective) was raised as a key concern.16  As Ess (2005)  notes, the terms 'Eastern' and 'Western' are not unproblematic, and are more products of colonialization than accurate terms for diverse nations and cultures. However, the terms continue to be used widely in the literature as shorthand for many of the broad cultural differences that are relevant to this paper; we therefore use the terms while being cognisant of their limitations.Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 'Plenty of people talk about the threat from AI; we want to be the threat.' US Deputy Secretary of Defence Patrick Shanahan in an email to Department of Defence employees (Houser 2018)  18 At a security forum in Washington D.C. in 2019, Kiron Skinner, the director of policy planning at the State Department, was quoted as saying about China 'This is a fight with a really different civilization and a different ideology and the United States has not had that before' and 'It's the first time that we will have a great power competitor that is not Caucasian' (Gehrke 2019) .19  Similarly, a range of challenges in digital security, a domain likely to be affected byAI (Brundage et al.  2018), are likely to be greatly exacerbated in the absence of agreed upon international cybersecurity practices and standards (UN General Assembly 2015).
			 http://www.worldvaluessurvey.org/wvs.jsp 21 http://www.asianbarometer.org/ Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			  For example, Floridi et al., 2018 present a 'unified framework'; Awad et al. 2018  talk about 'developing global, socially acceptable principles for machine ethics'; Jobin et al. 2019 set out to investigate 'global agreement' on what constitutes ethical AI. Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 Some excellent work on this question and related topics already exists in the field of intercultural information ethics-see for example Capurro (2005 Capurro ( , 2008 , Ess (2006) , Hongladarom et al. (2009) -but we would be particularly keen to see cross-cultural research collaborations around these topics, and greater attention paid to this work in more practical AI ethics discussions and initiatives.
			 Several authors of this paper are part of an initiative that aims to support cross-cultural research of this nature between the UK and China: https://ai-ethics-and-governance.institute/ 30 This specific translation was also provided to us by participants in the July 2019 workshop. 31 https://openai.com/charter/ Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 There is also a role for industry groups to play in encouraging cross-cultural collaboration through international workshops and conferences. We highlight the work of the AI Industry Alliance as an example of a recently established initiative in this space http://www.aiiaorg.cn/33  The Tianxia Fellowship, established by the Center for Long-term Priorities (http://www.longtermpriorities. org/), also includes AI safety and governance among other topics.
			 S. S. ÓhÉigeartaigh et al.


Abstract: In the late 2010s, various international committees, expert groups, and national strategy boards have voiced the demand to 'open' the algorithmic black box, to audit, expound, and demystify artificial intelligence. The opening of the algorithmic black box, however, cannot be seen only as an engineering challenge. In this article, I argue that only the sort of transparency that arises from critique-a method of theoretical examination that, by revealing pre-existing power structures, aims to challenge them-can help us produce technological systems that are less deceptive and more just. I relate the question of AI transparency to the broader challenge of responsible making, contending that future action must aim to systematically reconcile design-as a way of concealing-with critique-as a manner of revealing.

AI transparency: a matter of reconciling design with critique

In the age of ubiquitous computing, we are surrounded by objects that incorporate artificial intelligence solutions. We interact with different kinds of AI without realizing itusing online banking systems, searching for YouTube clips, or consuming news through social media-not really knowing how and when AI systems operate. Corporate strategies of secrecy and user interfaces that hide traces of AI-driven personalization combine with the inherent opacity of deep learning algorithms (whose inner workings are not directly comprehensible to human interpreters) to create a marked lack of transparency associated with all aspects of emerging technologies. It is in response to the widespread application of AI-based solutions to various products and services in the late 2010s that multiple expert groups-both national and international-have voiced the demand to 'open' the algorithmic black box, to audit, expound, and demystify AI. They claim that to ensure that the use of AI is ethical, we must design emerging systems to be transparent, explainable, and auditable.  1  The opening of the algorithmic black box, however, cannot be seen only as an engineering challenge. It is critique, as the underside of making, that prioritizes unboxing, debunking the illusion, seeing through-to reveal how an object really works. Critique-grounded in the tradition of Critical Theory and practiced by cultural studies, critical race theory, queer theory, as well as decolonial theory scholars, among others-moves beyond the technical detail to uncover the desires, ideologies, and social relations forged into objects, opening the black boxes of history, culture, and progress. In what follows, I argue that the calls for technological transparency demand that we combine the practice of design with critique. I relate the question of AI transparency to the broader challenge of responsible making, contending that future action must aim to systematically reconcile design-as a way of concealing-with critique-as a manner of revealing.
 Levels of opacity Many companies sell simple analytics tools as artificial intelligence, as something that supposedly supplants human intelligence to deliver better results. What is advertised as an 'AI solution' often relies on simple data analysis performed by human analysts. The use of metaphors and simplifications obfuscates human labor, labor that is outsourced, hidden away, in an invisible and immaterial factory, in a different part of the globe. According to Ian  Bogost (2015) , the 'metaphor of mechanical automation' is nothing more than a welldirected, but misleading, masquerade (n. pag.). Although the metaphor is only an approximation, a distortion, or even a 'caricature,' it convincingly plays the role of an accurate depiction of the whole. The term artificial intelligence, elusive, misleading and with a definition that has changed over time, forms the basis of the marketing stunt. Technology companies rely on abstract, overly schematic representations to simplify reality and arrive at an easily digestible, prepacked idea of the object, one that misrepresents the object's essence and overlooks its true composition, but also satiates the end user's curiosity. Other systems, as a matter of deliberate practice, incorporate complex data processing and machine learning surreptitiously. Shoshana  Zuboff (2019)  observes that the influence these systems have on our decision-making is 'designed to be unknowable to us  ' (p.11) ; that company strategies of misdirection serve as 'the moat that surrounds the castle and secures the action within' (p.65.)-a way for corporations like Google or Facebook to protect their secrets and mislead the public. These systems could, in theory, be designed to be more 'knowable.' Elements of the user interface could, for example, flag up when algorithmic operations are influencing the user's decision-making. Just like labels inform the consumer about the product's contents, the interfaces of Facebook and YouTube could announce to users that the information delivered by the platforms is algorithmically curated. Considering that only 24% of US college students realize Facebook automatically prioritizes certain posts and hides others  (Powers 2017) , such a feature would definitely be relevant. But the challenge of transparency at a time of unprecedented technological complexity cannot be approached only as a matter of failed (or indeed successful, depending on your position) communication. The 'opacity' of machine learning algorithms refers, after all, not only to 'institutional self-protection and concealment,' but also, as  Burrell (2016)  points out, to 'the mismatch between mathematical optimization in highdimensionality characteristic of machine learning and the demands of human-scale reasoning and styles of semantic interpretation' (p.2). The fundamental lack of transparency of systems that incorporate AI solutions relates not only to convoluted storytelling devised by marketing teams, or misleading interfaces and user experience design, but also-and most importantly-an emerging form of making brought about by the automation of cognitive tasks themselves. Considering this level of opacity of AI systems, a team of researchers from MIT's Intelligence and Decision Technologies Group developed a neural network named, suggestively, Transparency by Design Network  (Mascharka et al. 2018)  that not only performs 'human-like reasoning steps' to answer questions about the contents of images, but also 'visually renders its thought process' as it solves problems, allowing human analysts to interpret its decision-making. It is a deep learning system carrying out a sort of unboxing on itself, incorporating explainability in its very operations: a realization of the most common understanding of 'transparency' in the context of contemporary AI research.
 Transparency and critique The challenge of technological transparency that has now attracted the attention of policymakers has constituted a concern for cultural studies scholars, media theorists, and design philosophers for decades. In the early 1990s, the philosopher Paul Virilio (1994) noted that once 'synthetic vision' becomes a reality and the human subject is excluded from the process of observation of images 'created by the machine for the machine,' the power of statistical science to appear objective (and thus persuasive) will be 'considerably enhanced, along with its discrimination capacities' (p.75). Another prominent media theorist Friedrich  Kittler (2014)  expressed concerns about modern media technologies that are 'fundamentally arranged to undermine sensory perception.' Kittler wrote about a 'system of secrecy' based on user interfaces that 'conceal operations necessary for programming' and thus 'deprive users of the machine as a whole,' to suggest that perceiving the imitation, penetrating through the illusion of software 'that appears to be human,' is a fundamental challenge in the age of global-scale computing  (pp. 221-24) . Transparency, as Adrian Weller (2017) has poignantly noted, is an ambiguous term that will mean different things to different people. For the user, a transparent system will 'provide a sense for what [it] is doing and why,' while for an expert or regulator, it will enable auditing 'a prediction or decision trail in detail' (p.3). The calls for technological transparency have thus been filtered down to reach various stakeholder groups with different meanings and representing different interests. But what seems consistent in all of Footnote 1 (continued) 2019). Another interdisciplinary group of experts working under the auspices of IEEE (the world's largest professional association for electronic and electrical engineers) published, in 2019, the first edition of Ethically Aligned Design, a set of actionable recommendations on how to align design practices with society's values and principles, stressing that the 'standards of transparency, competence, accountability, and evidence of effectiveness should govern the development of autonomous and intelligent systems' (p. 5). these diverse takes on the development, use, and regulation of technology is that transparency is framed as a matter of design. In what follows, I problematize this claim, arguing that design, in the most fundamental sense, relies on concealment and obfuscation. I contend that only the sort of transparency that arises from critique-a method of theoretical examination that, by revealing pre-existing power structures, aims to challenge them-can help us produce technological systems that are less deceptive and more just.
 Design as blackboxing
 Art and artifice Coined in 1956 by John McCarthy, the term 'artificial intelligence' had its critics among those who attended the Dartmouth Conference (which famously established the field of AI); Arthur Samuel argued that 'the word artificial makes you think there's something kind of phony about this,  […]  or else it sounds like it's all artificial and there's nothing real about this work at all  ' (in: McCorduck 2004, p. 115) . The historian of AI Pamela McCorduck notes that while other terms, such as 'complex information processing,' were also proposed, it was 'artificial intelligence' that endured the trial of time. According to her, it is 'a wonderfully appropriate name, connoting a link between art and science that as a field AI indeed represents' (p. 115). She is referring indirectly here to the origins of the word artificial; in Latin, artificialis means 'of or belonging to art,' while artificium is simply a work of art, but also a skill, theory, or system. When the philosopher Vilém Flusser traced the etymology of the word 'design' in his The Shape of Things: A Philosophy of  Design (1999) , he referred to this relationship between art and artifice to argue that all human production, all culture, can be defined as a form of trickery. Flusser rejects the distinction between art and technology, and goes back to these ancient roots: the Greek for 'trap' is mechos (mechanics, machine); the Greek techne corresponds to the Latin ars; an artifex means a craftsman or artist, but also a schemer or trickster-to demonstrate that in their essence all forms of making are meant to help us 'elude our circumstances,' to cheat our own nature. Culture itself becomes a delusion brought about by means of design-a form of selfdeception that makes us believe we can free ourselves from natural restrictions by producing a world of artifice. From doors to rockets, from tents to computer screens, from pencils to mechanized intelligences, Flusser selects his examples to show that, ultimately, any involvement with culture is based on deception: sometimes 'this machine, this design, this art, this technology is intended to cheat gravity, to fool the laws of nature' (ch.1, n. pag.)-and sometimes to trick ourselves into thinking we control both gravity and the laws of nature. In that sense, art and technology are representative of the same worldview in which cultural production must be deceptive/artful enough to enable humans to go beyond the limits of what is (humanly) possible. Flusser refers to the act of weaving to explain the 'conspiratorial, even deceitful' (ch.18) character of design. In the process of carpet production, he points out, knotting is meant to deny its own warp, to hide the threads behind a pattern, so that anyone stepping on the finished rug perceives it as a uniform surface, according to the designer's plan. He offers weaving as one of the primordial forms of cultural production to embody trickery, but the same holds true for any form of design. The trick is always based on misdirection, shifting the end user's attention from the material to the application, from the current state of things to emerging possibilities and new futures. Designing is a methodical way of crafting alternative realities out of existing materials-a process of casting the intended shape onto the chosen fabric so as to create a new possibility. The material used in that process must, so to speak, dematerialize: it has to disappear from view and give way to the new object-to abstract the end result from the point of origin and the labor process. By obfuscating some components while exhibiting others, 'ideal' design enables an end user's cognitive efficiency.
 Patterns, layers, and repetitions For Flusser, any product of human making is both an object and an obstacle-Latin objectum, Greek problema-or, more specifically, any object is also an 'obstacle that is used for removal of obstacles' (ch.9). To move forward, we solve problems that lie ahead and obstruct our way; we produce objects that help us remove these obstacles; but the same objects turn into obstacles for those that come after us. In other words, since the results of human problem-solving are stored in objects, progress involves obfuscation and forgetting. We come up with a solution and, with time, this singular idea turns into a pattern; others use the already established template to produce new, more complex structures and these structures turn into new patterns, covering up previous layers of design with new design. To expedite the process of production, to advance, to move faster, the designer turns to these conventions and templates, choosing from a menu of preprogrammed options-or abstracting new rules based on previous patterns. And as the complexity of the production process increases, the reliance on patterns grows too. New design always depends on previous design, and this ultimate dependence on patterns and abstractions complicates understanding the process in its totality. In the age of ubiquitous computing, speaking of obfuscation by design becomes of particular importance. In 2015, Benjamin Bratton called his model of the new kind of layering brought about by planetary-scale computation 'the Stack': 'an accidental megastructure, one that we are building both deliberately and unwittingly and is in turn building us in its own image' (p.5). New technologies 'align, layer by layer, into something like a vast, if also incomplete, pervasive if also irregular, software and hardware Stack' (p.5). This makes it hard to perceive the Stack's overarching structure, indeed, to see it as design, however incidental. Today, we produce new technologies, new objects, to see, know, and feel more, to register what is normally hidden from our view, meanwhile, creating complex systems based on multiple, invisible layers and algorithmic operations whose effects are not always comprehensible even to the designers themselves.
 Automations and automatisms In her comprehensive account of what she calls 'surveillance capitalism,' Shoshana Zuboff points out the dangers of technological illusion-'an enduring theme of social thought, as old as the Trojan horse' (p.16)-that serves the new economic project in rendering its influence invisible. Surveillance capitalism claims 'human experience as free raw material for translation into behavioral data,' and turns that data into 'prediction products' that can later be sold to advertisers (p.8). Echoing the work of philosophers such as Bernard  Stiegler (2014 Stiegler ( , 2015  or Antoinette  Rouvroy (2016) , Zuboff argues that the ultimate goal of this new form of capitalism is 'to automate us,' by reprogramming our behavior and desires. Various internet platforms that dominate the market prompt us to action, influence our decision making, relying on big data analyses of our preference trends online. Automated systems create statistical models to profile users, tracing any emerging patterns in countless interactions with digital products; patterns turn into further abstractions, new models that are later reflected in new products and solutions, which end up 'automating' us, guiding our decision-making without our knowing. But is this process specific to AI-enhanced personalization under surveillance capitalism? Bratton has recently argued that what 'at first glance looks autonomous (self-governing, set apart, able to decide on its own) is, upon closer inspection, always also decided in advance by remote ancestral agents and relays, and is thus automated as well  ' (2019, loc.345, n. pag.) . Any decision taken now relies on multiple decisions taken in the past; new design depends on previous design; a new object coalesces from an aggregation of old solutions. Culture is an amalgamation of such objectsobjects that, ironically, become obstacles because they are meant to enable our cognitive efficiency. A tool becomes an obstacle because the results of our problem-solving and labor are already stored within it; a tool must never be seen as a tool, as its use must be intuitive-it must remain imperceptible; any new tool meant to advance the process is made with existing tools, and so the emerging layering of design in the Anthropocene makes it harder to distinguish between tool and fabric. Extending this to the ongoing automation of cognitive tasks in the age of ubiquitous computing, the phenomenon takes on new scale. This is why the emerging need for transparency refers not so much to company politics of disinformation or algorithmic black boxes, as to the very essence of our culture, as a process of knowledge production, pattern formation, and concealment. Particular problems caused by the widespread adoption of automated decision-making systems, such as algorithmic bias, can have specific, targeted, solutions in the form of new policy, engineering standards, or better education. But a shift of focus from the particular to the total is more than an exercise in theory-it makes us realize that transparency has never been at the heart of our making, that design has always been a form of blackboxing. There is, in that sense, something deeply anti-cultural about transparency. Or, putting it differently, there is nothing natural about transparency by design: we have been programmed to cover up as we make, not the opposite. The ongoing transformation of lived experiences into data is a new analytical paradigm that demands our intervention, truly calls for an 'unboxing,' an excavation of processes and data trails. But the opening of the algorithmic black box cannot be viewed only as a technical issue-precisely because any solution is, first and foremost, a result of cultural blackboxing. While contemporary debates on AI focus on transparency as a direct response to the opacity of algorithms, what we are in need of are approaches that aim to 'unbox' new technologies as objects-obstacles, solutions that aim towards cognitive automation, products that store the results of problem-solving performed 'by remote ancestral agents,' and that can thus perpetuate injustices via automatically accepted patterns and norms.
 Critique as unboxing
 Apparent transparencies Among entries on subjects such as theology, economics, and medicine, Denis Diderot and Jean le Rond d'Alembert included in their Dictionary of the Sciences, Arts, and Crafts, entries on artisanal practices that detail the individual steps in the processes of production adopted in clockmaking, tailoring, woodworking, and many others. One such entry focuses on the making of artificial flowers: the first plate (Fig.  1 ) depicts a dozen workers scattered across the main workshop area, performing different tasks at various stages of manufacturing, while following pages of illustrations showcase the most popular templates used to emboss specific petal shapes onto fabric, with a final plate celebrating the finished commodity. By bringing to view the backstages of production, the Encyclopedia was essentially undesigning, reversing the process of 'conspiratorial weaving' described by Flusser. Now, in an age of growing technological complexity, shaped by significant degrees of cognitive automation, there is a need for a similar undesigning of new technologies. The artist Todd McLellan's photographs (Fig.  2 ) that document his multiple attempts at taking various objects apart are a suggestive illustration of this challenge in the age of extreme technological complexity. We might dissemble our smartphone, but learning what is hidden beneath the interactive surface of the touchscreen will never give us an indication of how the device really works and, more significantly, in whose interest. The meaningless innards of the device become symbolic of the contestable quality that transparency really is-if we think of it as a condition for, or indeed a guarantee of, understanding. Critical undesigning cannot be confused with a simple act of reverse engineering. There can be transparency without critique, or apparent transparency: but a sort of transparency that does not arise from critical processes of unboxing is unlikely to advance comprehension. In his lecture on black boxes, Galloway (2010) relates Marx's idea of descent into 'the hidden abode of production' (p.7), as a means of uncovering capital relations forged into commodities, to 'traditions of critical inquiry, in which objects were unveiled or denaturalized to reveal their inner workings-from Descartes's treatise on method […] to the Freudian plumbing of the ego' (p.5). Based on the assumption that the surface is merely a superficial facade to be penetrated by means of critique, these theories prioritized the interior and perceived objects as 'mystical black boxes waiting to be deciphered to reveal the rationality (of history, of totality) harbored within' (p.3). For the purpose of this article, critique is understood as a broad set of methodologies, grounded in the tradition of Critical Theory, that perform a metaphorical dismantling of objects to reveal how hidden and immaterial layers of design reflect social and economic structures-and how the power relations these structures generate become the sources of injustice, oppression, and exploitation. Critique's ultimate goal is to uncover and challenge the system(s) that objects of design engender; revelation is conceived of as the condition necessary for resistance-and systemic transformation. Looking beyond individual design flaws (and fixes), critique points to those 'ancestral relays' that automate our thinking-to patterns, repetitions, and automatisms so deeply ingrained, weaved into the fabric of our culture, that they become imperceptible-in particular to those who do not experience the injustices resulting from the adoption of already established patterns.
 Critique in the age of AI A former YouTube employee, Guillaume Chaslot, has coined the term Algotransparency to describe an experiment in which he investigates the terms appearing most frequently in the titles of videos recommended by YouTube. A program developed by Chaslot and his team traces thematic patterns in YouTube recommendations to prove there exists a systemic bias that promotes controversial clips. His research suggests Google's platform indeed 'systematically amplifies videos that are divisive, sensational and conspiratorial'  (Lewis 2018) -that the recommendations are not related to the individual user's interests (as the company claims), but rather-exploit controversy to boost clickability. Algotransparency attempts to unbox the logic of YouTube's copyrightprotected recommendation algorithm without directly looking into the system's black box, concentrating only on the effects of its activity. This specific experiment gives a good indication of where we should be directing our attention: focusing not so much on how YouTube operates, as on why it works at all. This question extends beyond the technicality of the algorithm, to more widely interrogate the forces orchestrating our consumption of digital goods and whose interests they serve-what Zuboff calls surveillance capitalism, or what Stiegler refers to as hyperindustrialism. Ian Bogost (2015) has argued that the illusion of automation in technology-the trick that misdirects our attention from essential questions about human decision-making incorporated into emerging systems-breaks down 'once you bother to look at how even the simplest products are really produced.' In 2014 he collaborated with Alexis Madrigal to analyze Netflix's recommendation system and demonstrate that the platform's operations are distributed among so many different agents-including human curators who hand-tag all Netflix content-'that only a zealot would call the end result an algorithm'(Bogost 2015). Many experiments and critical projects try to achieve something similar: debunk the illusion of software by exposing AI as processual and collaborative, tracing the results of data analysis back to human decisions, biases, and labor. In their Anatomy of an AI System, for instance, Crawford and Joler (2018) present a figurative dissection of Amazon's Echo device that brings to view the invisible mechanisms and dynamisms that the product encapsulates (Fig.  3 ). The detailed mapping of various objects and agents, as well as multiple layers of interaction between those elements, constitutes a representation of the system as composed not only of hardware and software, data and computation, but also human labor and planetary resources. Critique in the age of extreme technological complexity is as much about dissecting and penetrating, as it is about charting the invisible and immaterial terrains of interaction, analysis, consumption, and computation; mapping wider relations between energies, influences, and resources under surveillance capitalismpatterns of exploitation of both people and environments that the production of objects/obstacles entails. In another project, Crawford teamed up with the artist Trevor Paglen to carry out what they call an archeology of datasets, such as ImageNet, used in machine learning to train AI to recognize specific elements of images-sets that can also become sources of bias inscribed into emerging systems. By excavating the datasets' underlying structures,  Crawford and Paglen (2019)  aim to reveal their implicit meanings: 'we have been digging through the material layers, cataloguing the principles and values by which something was constructed, and analyzing what normative patterns of life were assumed, supported, and reproduced.' The blackboxed Earth, a world deeply transformed by ubiquitous computing, by various layers of what Bratton calls the Stack, demands this form of unearthing. Successful critique in the age of AI exposes the technology as relying on human cognition and decision-making; more broadly, critique reveals the constellation of objects/obstacles as products of layered problem-solving, a flawed process that is necessarily tainted by pre-existing patterns and abstractions, biases and beliefs. Prioritizing reflection over efficiency, critique becomes a methodical way of resisting our reliance on patterns-patterns that allow us to move faster, but that can also harbor previously made assumptions-about gender (Costanza-Chock 2018) or race  (Benjamin 2019)  as particularly poignant examples-and thus perpetuate, rather than challenge, pre-existing forms of injustice.
 Reconciling design with critique In keeping up with the societal demand for transparent AI, the big players of the tech industry have been introducing changes in their engineering standards and organizational structures, hiring ethicists and policy specialists to cooperate with their product development teams. In 2016, Microsoft established the Aether Committee, a body of senior advisors to the company's leadership, that provides guidance 'on rising questions, challenges, and opportunities with the development and fielding of AI technologies' (2020), and oversees the work of other teams 'to ensure that AI products and services align with Microsoft's AI principles'-which include transparency and accountability. In 2017, DeepMind set up its Ethics and Society team 'to guide the responsible development and deployment of AI.' The team composed of 'ethicists and policy researchers' collaborates with the company's AI research team 'to understand how technical advances will impact society, and find ways to reduce risk.' Smaller industry players who decide to follow suit, but cannot afford to establish their own ethics 'departments,' begin to enlist the help of 'ethical auditing' companies; Glassbox, for example, is a tech consultancy startup, founded in 2018, that aims to 'provide clarity to the black box' by analyzing software products for signs of bias and training the client company's employees about systemic injustices. This way, elements of critique that reveal potential implications of human decision-making in design are supposed to become part of the production pipeline. These sites of interaction between 'humanists' and 'technologists' in the industry-even if, in some cases, they amount to nothing more than backdrops for press releasesdeserve our attention. Specifically, they require of us a comprehensive rethinking of what satisfies our desire for transparency in the age of extreme technological complexity. Can a system of checks and balances in the industry, an ongoing negotiation between blackboxing and unboxing, lead to anything more than design that anticipates critique? Is critique from within the industry necessarily a compromise and, therefore, nothing more than another step in the process of production of objects that are also obstacles? Must critique be external to the process of designing to remain genuine? If design is, fundamentally, blackboxing and automation, and critique is unboxing that aims to reverse the process of 'conspiratorial weaving,' then we could conclude that these two sides of human activity are in stark opposition to one another-that design is incompatible with critique. Instituting a real change in the way we move forward must focus precisely on tackling this ostensible impossibility-on reconciling design with critique, progress with suspension, production with reflection. The calls for transparency by design require that the 'making' itself be reinvented to incorporate critique. If a systemic transformation depends on a new-found compatibility between design and critique, then the designers of emerging systems should turn to already existing, alternative design practices to learn what combining the processes of making and critique might entail. Anthony Dunne and Fiona Raby, who have been pioneers in the field of 'critical design' since the late 1990s, have been advocating for design understood as 'critical thought translated to materiality  ' (2013, p.35 ). An object of design in their framing must become a critical challenge-as much for the designer, as for the user: 'it encourages people to question, in an imaginative, troubling, and thoughtful way, everydayness and how things could be different.' (p.189) More recently, Ratto (2011) coined the term 'critical making' to describe a design process that focuses on 'the act of shared construction itself as an activity and a site for enhancing and extending conceptual understandings of critical sociotechnical issues' (p.254)-with those who take part as the agents of critique. For Dunne and Raby, design has become 'so absorbed in industry, so familiar with dreams of industry, that it is almost impossible to dream its own dreams' (p.88). The suggestion is that the challenge lies not so much in the incompatibility between making and critique, as between the futures imagined by the industry and the dreams functioning outside of it. While elements of critique-gender critique and critical race theory in particular-seem to have already penetrated sections of the industry in the form of the mentioned ethics auditing services, reconciling the critique of surveillance capitalism and hyperindustrialism, the forces behind most of today's innovation, with the design of new technologies within corporate structures appears a considerable (and counterintuitive) feat. Perhaps this is where we should direct our attention: more research is needed on how practices such as critical design and critical making can influence the process of AI design; how critique can be operationalized within the industry to challenge industrial values and visons, including the idea of 'progress' itself. In a recently published volume on the practice of 'undesign' (McNamara and Coombs 2019), Cameron Tonkinwise's essay proposes what he calls 'anti-progressive' design as a means of interrogating the designers' internalized desire for 'progress.' While users, as he rightly observes, are willing 'to unlearn and relearn modes of interaction' if what is new is also 'easier and more convenient, and hopefully more effective and pleasurable' (p.76), it is the designers who should learn how 'not to prefer progress, or how to prefer what does not feel like progress' (p.81). Tonkinwise argues it is the designers' duty 'to find a way to pursue the destructively preferable without casting the resulting change as progress: what is preferable are futures that no longer appear to be mere advancements of what currently exists  ' (p.81) . This is to say that the responsibility for future action lies, primarily, with the designers: the human makers who, as products of a specific culture, are being increasingly challenged to become aware of their own biases and automatisms that predetermine their actions and choices. For designers, combining design with critique, rather than an attempt at making things transparent, would constitute an attempt at becoming self-conscious. Flusser argued that a renewed form of culture would have to be a culture 'aware of the fact that it was deceptive' (ch.1). To rethink technological transparency, we should first recognize that the designer has always been a trickster laying out traps, technologizing misdirection to pave the way forward. All aspects of design-practical, political, moral-would have to reflect this awareness: as we move forward, we must acknowledge that any problem solved now will also form a trap for those coming after us. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/. Fig. 1 Fig. 2 12 Fig. 1 Maker of artificial flowers, 1765 (https ://hdl.handl e.net/2027/ spo.did22 22.0001.451)
 Fig. 3 3 Fig. 3 Kate Crawford and Vladan Joler, Anatomy of an AI System, 2018 (courtesy of the artists)
			 The High-Level Expert Group on AI convened by the European Commission presented its Ethics Guidelines for Trustworthy Artificial Intelligence in the early 2019. The document identifies several key characteristics of a system that can be deemed trustworthy, which include transparency, defined as a form of traceability of data, operations, and business models that shape the end product (AI HLEG