Abstract: Much of estimation of human internal state (goal, intentions, activities, preferences, etc.) is passive: an algorithm observes human actions and updates its estimate of human state. In this work, we embrace the fact that robot actions affect what humans do, and leverage it to improve state estimation. We enable robots to do active information gathering, by planning actions that probe the user in order to clarify their internal state. For instance, an autonomous car will plan to nudge into a human driver's lane to test their driving style. Results in simulation and in a user study suggest that active information gathering significantly outperforms passive state estimation.

Dorsa Sadigh
S Shankar
Sastry Sanjit
A Seshia
Anca Dragan

Information Gathering Actions over Human Internal State

I. introduction Imagine driving on the highway. Another driver is in the lane next to you, and you need to switch lanes. Some drivers are aggressive and they will never brake to let you in. Others are more defensive and would gladly make space for you. You don't know what kind of driver this is, so you decide to gently nudge in towards the other lane to test their reaction. At an intersection, you might nudge in to test if the other driver is distracted and they might just let you go through (Fig.  1  bottom left). Our goal in this work is to give robots the capability to plan such actions as well. In general, human behavior is affected by internal states that a robot would not have direct access to: intentions, goals, preferences, objectives, driving style, etc. Work in robotics and perception has focused thus far on estimating these internal states by providing algorithms with observations of humans acting, be it intent prediction  [23] ,  [13] ,  [6] ,  [3] ,  [4] ,  [16] , Inverse Reinforcement Learning  [1] ,  [12] ,  [15] ,  [22] ,  [18] , driver style prediction  [11] , affective state prediction  [10] , or activity recognition  [20] . Human state estimation has also been studied in the context of human-robot interaction tasks. Here, the robot's reward function depends (directly or indirectly) on the human internal state, e.g., on whether the robot is able to adapt to the human's plan or preferences. Work in assistive teleoperation or in human assistance has cast this problem as a Partially Observable Markov Decision Process, in which the robot does observe the physical state of the world but not the human internal state -that it has to estimate from human actions. Because POMDP solvers are not computationally efficient, the solutions proposed thus far use the current estimate Fig.  1 : We enable robots to generate actions that actively probe humans in order to find out their internal state. We apply this to autonomous driving. In this example, the robot car (yellow) decides to inch forward in order to test whether the human driver (white) is attentive. The robot expects drastically different reactions to this action (bottom right shows attentive driver reaction in light orange, and distracted driver reaction in dark orange). We conduct a user study in which we let drivers pay attention or distract them with cellphones in order to put this state estimation algorithm to the test. of the internal state to plan (either using the most likely estimate, or the entire current belief), and adjust this estimate at every step  [8] ,  [7] ,  [5] . Although efficient, these approximations sacrifice an important aspect of POMDPs: the ability to actively gather information. Our key insight is that robots can leverage their own actions to help estimation of human internal state. Rather than relying on passive observations, robots can actually account for the fact that humans will react to their actions: they can use this knowledge to select actions that will trigger human reactions which in turn will clarify the internal state. We make two contributions: An Algorithm for Active Information Gathering over Human Internal State. We introduce an algorithm for planning robot actions that have high expected information gain. Our algorithm uses a rewardmaximization model of how humans plan their actions in response to those of the robot's  [19] , and leverages the fact that different human internal states will lead to different human reactions to speed up estimation. Fig.  1  shows an example of the anticipated difference in reaction between a distracted and an attentive driver. Application to Driver Style Estimation. We apply our algorithm to estimating a human driver's style during the interaction of an autonomous vehicle with a human-driven vehicle. Results in simulation as well as from a user study suggest that our algorithm's ability to leverage robot actions for estimation leads to significantly higher accuracy in identifying the correct human internal state. The autonomous car plans actions like inching forward at an intersection (Fig.  1 ), nudging into another car's lane, or braking slightly in front of a human-driven car, all to estimate whether the human driver is attentive. Overall, we are excited to have taken a step towards giving robots the ability to actively probe end-users through their actions in order to better estimate their goals, preferences, styles, and so on. Even though we chose driving as our application domain in this paper, our algorithm is general across different domains and types of human internal state. We anticipate that applying it in the context of human goal inference during shared autonomy, for instance, will lead to the robot purposefully committing to a particular goal in order to trigger a reaction from the user, either positive or negative, in order to clarify the desired goal. Of course, further work is needed in order to evaluate how acceptant end-users are of different kinds of such probing actions.
 II. Information Gathering Actions We start with a general formulation of the problem of a robot needing to maximize its reward by acting in an environment where a human is also acting. The human is choosing its actions in a manner that is responsive to the robot's actions, and also influenced by some internal variables. While most methods that have addressed such problems have proposed approximations based on passively estimating the internal variable and exploiting that estimate, here we propose a method for active information gathering that enables the robot to purposefully take actions that probe the human. Finally, we discuss our implementation in practice, which trades off between exploration and exploitation.
 A. General Formulation We define a human-robot system in which the human's actions depend on some human internal state ϕ that the robot does not directly observe. In a driving scenario, ϕ might correspond to driving style: aggressive or timid, attentive or distracted. In collaborative manipulation scenarios, ϕ might correspond to the human's current goal, or their preference about the task. We let x ∈ X be a continuous physical state of our system. For our running example of autonomous cars, this includes position, velocity and heading of the autonomous and human driven vehicles. Let ϕ ∈ Φ be the hidden variable, e.g., human driver's driving style. We assume the robot observes the current physical state x t , but not the human internal state ϕ. The robot and human can apply continuous controls u R and u H . The dynamics of the system evolves as robot's and human's control inputs arrive at each step: x t+1 = f H ( f R (x t , u t R ), u t H ). (1) Here, f R and f H represent how the actions of the robot and of the human respectively affect the dynamics, and can be applied synchronously or asynchronously. We assume that while x changes via (1) based on the human and robot actions, ϕ does not. For instance, we assume that the human maintains their preferences or driving style throughout the interaction. The robot's reward function in the task depends on the current state, the robot's action, as well as the action that the human takes at that step in response, r R (x t , u t R , u t H ). If the robot has access to the human's policy π H (x, u R , ϕ), maximizing robot reward can be modeled as a POMDP  [9]  with states (x, ϕ), actions u R , and reward r R . In this POMDP, the dynamics model can be computed directly from (1) with u t H = π H (x t , u t R , ϕ). The human's actions can serve as observations of ϕ via some P(u H |x, u R ). In Sec. II-C we introduce a model for both π H and P(u H |x, u R , ϕ) based on the assumption that the human is maximizing their own reward function. If we were able to solve the POMDP, the robot would estimate ϕ based on the human's actions, and optimally trade off between exploiting its current belief over ϕ, and actively taking information gathering actions meant to cause human reactions that give the robot a better estimate of the hidden variable ϕ. Because POMDPs cannot be solved tractably, several approximations have been proposed for similar problem formulations  [8] ,  [11] ,  [7] . These approximations are passively estimating the human internal state, and exploiting the belief to plan robot actions.  1  In this work, we take the opposite approach: we focus explicitly on active information gathering. We enable the robot to decide to actively probe the person to get a better estimate of ϕ. Our method can be leveraged in conjunction with exploitation methods, or be used alone when human state estimation is robot's primary objective.
 B. Reduction to Information Gathering At every step, the robot can update its belief over ϕ via: b t+1 (ϕ) ∝ b t (ϕ) • P(u H |x t , u R , ϕ). ( 2 ) To explicitly focus on taking actions to estimate ϕ, we redefine the robot's reward function to capture the information gain at every step: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) (3) with H(b) being the entropy over the belief: H(b) = − ∑ ϕ b(ϕ) log(b(ϕ)) ∑ ϕ b(ϕ) . ( 4 ) Optimizing expected reward now entails reasoning about the effects that the robot actions will have on what observations the robot will get, i.e., the actions that the human will take in response, and how useful these observations will be in shattering ambiguity about ϕ.
 C. Solution: Human Model & Model Predictive Control We solve the information gathering planning problem via Model Predictive Control (MPC)  [14] . At every time step, we find the optimal actions of the robot u * R by maximizing the expected reward function over a finite horizon. Notation. Let x 0 be the state at the current time step, i.e., at the beginning of the horizon. u R = (u 0 R , . . . , u N−1 R ) be a finite sequence of the robot's continuous actions, and u H = (u 0 H , . . . , u N−1 H ) be a finite sequence of human's continuous actions. Further, let R R (x 0 , u R , u H ) denote the reward over the finite horizon, if the agents started in x 0 and executed u R and u H , which can be computed via the dynamics in  (1) . MPC Maximization Objective. At every time step, the robot is computing the best actions for the horizon: u * R = arg max u R E ϕ R R (x 0 , u R , u * ϕ H (x 0 , u R )) (5) where u * ϕ H (x 0 , u R ) corresponds to the actions the human would take from state x 0 over the horizon of N steps if the robot executed actions u R . Here, the expectation is taken over the current belief over ϕ, b 0 . Simplifying (5) using the definition of reward from (3), we get: u * R = arg max u R E ϕ H(b 0 ) − H(b N ) , (6) u * R = arg max u R E ϕ − H(b N ) , (7) where the expectation remains with respect to b 0 . Human Model. We assume that the human maximizes their own reward function at every step. We let r ϕ H (x t , u t R , u t H ) represent human's reward function at time t, which is parametrized by the human internal state ϕ. Then, the sum of human rewards over horizon N is: R ϕ H (x 0 , u R , u H ) = N−1 ∑ t=0 r ϕ H (x t , u t R , u t H ) (8) Building on our previous work  [19] , which showed how the robot can plan using such a reward function when there are no hidden variables, we compute u * ϕ H (x 0 , u R ) through an approximation. We model the human as having access to u R a priori, and compute the finite horizon human actions that maximize the human's reward: u * ϕ H (x 0 , u R ) = arg max u H R ϕ H (x 0 , u R , u H ) (9) One can find r ϕ H through Inverse Reinforcement Learning (IRL)  [1] ,  [12] ,  [15] ,  [22] , by getting demonstrations of human behavior associated with a direct measurement of ϕ. The robot can use this reward in the dynamics model in order to compute human actions via  (9) . To update the belief b and compute expected reward in  (7) , we still need an observation model. We assume that actions with lower reward are exponentially less likely, building on the principle of maximum entropy  [22] : P(u H |x, u R , ϕ) ∝ exp(ϕ H (x, u R , u H )) (10) Optimization Procedure. To solve (5) (or equivalently (  7 )), we use a gradient descent optimization method, L-BFGS, designed for unconstrained nonlinear problems  [2] . Therefore, we would like to find the gradient of the objective in equation (  5 ) with respect to u R . Since the objective is the expectation of R R , we can reformulate this gradient as: ∂E ϕ R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R = ∑ ϕ ∂R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R • b 0 (ϕ) (11) Then, we only need to find ∂R R ∂u R , which is equivalent to: ∂R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R = ∂R R (x 0 , u R , u H ) ∂u H ∂u * ϕ H ∂u R + ∂R R (x 0 , u R , u H ) ∂u R | u H =u * ϕ H (x 0 ,u R ) (12 ) Because R R , as indicated by  (7) , simplifies to the negative entropy of the updated belief, we can compute both ∂R R (x 0 ,u R ,u H ) ∂u H and ∂R R (x 0 ,u R ,u H ) ∂u R | u H =u * ϕ H (x 0 ,u R ) symbolically. This leaves ∂u * ϕ H ∂u R . We use the fact that the gradient of R H will evaluate to zero at u * ϕ H : ∂R H ∂u H (x 0 , u R , u * ϕ H (x 0 , u R )) = 0 (13) Now, differentiating this expression with respect to u R will result in: ∂ 2 R H ∂u 2 H ∂u * ϕ H ∂u R + ∂ 2 R H ∂u H ∂u R ∂u R ∂u R = 0 (14) Then, solving for ∂u * ϕ H ∂u R enables us to find the following symbolic expression: ∂u * ϕ H ∂u R = [− ∂ 2 R H ∂u H ∂u R ][ ∂ 2 R H ∂u 2 H ] −1 . ( 15 ) This expression allows finding a symbolic expression for the gradient in equation  (11) .
 D. Explore-Exploit Trade-Off In practice, we use information gathering in conjunction with exploitation. We do not solely optimize the reward from Sec. II-B, but optimize it in conjunction with the robot's actual reward function assuming the current estimate of ϕ: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) ( 16 ) + λ • r goal (x t , u R , u H , b t ) At the very least, we do this as a measure of safety, e.g., we want an autonomous car to keep avoiding collisions even when it is actively probing a human driver to test their reactions. We choose λ experimentally, though existing techniques that can better adapt λ over time  [21] . Despite optimizing this trade-off, we do not claim that our method as-is can better solve the general POMDP formulation from Sec. II-A: only that it can be used to get better estimates of human internal state. The next sections test this in simulation and in practice, in a user study, and future work will look at how to leverage this ability to better solve human-robot interaction problems.
 III. Simulation Results In this section, we show simulation results that use the method from the previous section to estimate human driver type in the interaction between an autonomous vehicle and a human-driven vehicle. In this section, we consider three different autonomous driving scenarios. In these scenarios, the human is either distracted or attentive during different driving experiments. The scenarios are shown in Fig.  2 , where the yellow car is the autonomous vehicle, and the white car is the human driven vehicle. Our goal is to plan to actively estimate the human's driving style in each one of these scenarios, by using the robot's actions.
 A. Attentive vs. Distracted Human Driver Models Our technique requires reward functions r ϕ H that model the human behavior for a particular internal state ϕ. We obtain a generic driver model via Continuous Inverse Optimal Control with Locally Optimal Examples  [12]  from demonstrated trajectories in a driving simulator in an environment with multiple autonomous cars, which followed precomputed routes. We parametrize the human reward function as a linear combination of features, and learn weights on the features. We use various features including features for bounds on the control inputs, features that keep the vehicles within the road boundaries and close to the center of their lanes. Further, we use quadratic functions of speed to capture reaching the goal, and Gaussians around other vehicles on the road to enforce collision avoidance as part of the feature set. We then adjust the learned weights to model attentive vs. distractive drivers. Specifically, we modify the weights of the collision avoidance features, so the distracted human model has less weight for these features. Therefore, the distracted driver is more likely to collide with the other cars while the attentive driver has high weights for the collision avoidance feature.
 B. Manipulated Factors We manipulate the reward function that the robot is optimizing. In the passive condition, the robot optimizes a simple reward function for collision avoidance based on the current belief estimate. It then updates this belief passively, by observing the outcomes of its actions at every time step. In the active condition, the robot trades off between this reward function and the Information Gain from (3) in order to explore the human's driving style. We also manipulate the human internal state to be attentive or distracted. The human is simulated to follow the ideal model of reward maximization for our two rewards.
 C. Driving Simulator We use a simple point-mass model for the dynamics of the vehicle, where x = x y θ v is the state of the vehicle. Here, x and y are the coordinates of the vehicle, θ is the heading, and v is the speed. Each vehicle has two control inputs u = u 1 u 2 , where u 1 is the steering input, and u 2 is acceleration. Further, we let α be a friction coefficient. Then, the dynamics of each vehicle is formalized as: [ ẋ ẏ θ v] = [v • cos(θ) v • sin(θ) v • u 1 u 2 − α • v]. (17)
 D. Scenarios and Qualitative Results Scenario 1: Nudging In to Explore on a Highway. In this scenario, we show an autonomous vehicle actively exploring the human's driving style in a highway driving setting. We contrast the two conditions in Fig.  2 (a). In the passive condition, the autonomous car drives on its own lane without interfering with the human throughout the experiment, and updates its belief based on passive observations gathered from the human car. However, in the active condition, the autonomous car actively probes the human by nudging into her lane in order to infer her driving style. An attentive human significantly slows down (timid driver) or speeds up (aggressive driver) to avoid the vehicle, while a distracted In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.  2(c ). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. Autonomous Vehicle Human Driven Vehicle Passive Estimation Active Info Gathering In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 E. Quantitative Results Throughout the remainder of the paper, we use a common color scheme to plot results for our experimental conditions. We show this common scheme in Fig.  3 : darker colors (black and red) correspond to attentive humans, and lighter colors (gray and orange) correspond to distracted humans. Further, the shades of orange correspond to active information gathering, while the shades of gray indicate passive information gathering. We also use solid lines for real users, and dotted lines for scenarios with an ideal user model learned through inverse reinforcement learning. This table is representative for the legends of Fig.  4 , Fig.  ? ?, Fig.  5 , and Fig.  6 . Fig.  4  plots, using dotted lines, the beliefs over time for the attentive (left) and distracted (right) conditions, comparing in each the passive (dotted black and gray respectively) with the active method (dotted dark orange and light orange respectively). In every situation, the active method achieves a more accurate belief (higher values for attentive on the left, when the true ϕ is attentive, and lower values on the right, when the true ϕ is distracted). In fact, passive estimation sometimes incorrectly classifies drivers as attentive when they are distracted and vice-versa. The same figure also shows (in solid lines) results from our user study of what happens when the robot no longer interacts with an ideal model. We discuss these in the next section. Fig.  5  and Fig.  6  plot the corresponding robot and human trajectories for each scenario. The important takeaway from these figures is that there tends to be a larger gap between attentive and distracted human trajectories in the active condition (orange shades) than in the passive condition (gray shades), especially in scenarios 2 and 3. It is this difference that helps the robot better estimate ϕ: the robot in the active condition is purposefully choosing actions that will lead to large differences in human reactions, in order to more easily determine the human driving style.
 IV. User Study In the previous section, we explored planning for an autonomous vehicle that actively probes a human's driving style, by braking or nudging in and expecting to cause reactions from the human driver that would be different depending on their style. We showed that active exploration does significantly better at distinguishing between attentive and distracted drivers using simulated (ideal) models of drivers. Here, we show the results of a user study that evaluates this active exploration for attentive and distracted human drivers.
 A. Experimental Design We use the same three scenarios discussed in the previous section. Manipulated Factors. We manipulated the same two factors as in our simulation experiments: the reward function that the robot is optimizing (whether it is optimizing its reward through passive state estimation, or whether it is trading off with active information gathering), and the human internal state (whether the user is attentive or distracted). We asked our users to pay attention to the road and avoid collisions for the attentive case, and asked our users to play a game on a mobile phone during the distracted driving experiments. Dependent Measure. We measured the probability that the robot assigned along the way to the human internal state. Hypothesis. The active condition will lead to more accurate human internal state estimation, regardless of the true human internal state. Subject Allocation. We recruited 8 participants (2 female, 6 male) in the age range of 21-26 years old. All participants owned a valid driver license and had at least 2 years of driving experience. We ran the experiments using a 2D driving simulator with the steering input and acceleration input provided through a steering wheel and a pedals as shown in Fig.  1 . We used a within-subject experiment design with counterbalanced ordering of the four conditions.
 B. Analysis We ran a factorial repeated-measures ANOVA on the probability assigned to "attentive", using reward (active vs passive) and human internal state (attentive vs distracted) as factors, and time and scenario as covariates. As a manipulation check, attentive drivers had significantly higher estimated probability of "attentive" associated than distracted drivers (.66 vs .34, F = 3080.3, p < .0001). More importantly, there was a signifiant interaction effect between the factors (F = 1444.8, p < .000). We ran a post-hoc analysis with Tukey HSD corrections for multiple comparisons, which showed all four conditions to be significantly different from each other, all contrasts with p < .0001. In particular, the active information gathering did end up with higher probability mass on "attentive" than the passive estimation for the attentive users, and lower probability mass for the distracted user. This supports our hypothesis that our method works, and active information gathering is better at identifying the correct state. Fig.  4  compares passive (grays and blacks) and active (light and dark oranges) across scenarios and for attentive (left) and distracted (right) users. It plots the probability of attentive over time, and the shaded regions correspond to standard error. From the first column, we can see that our algorithm in all cases detects human's attentiveness with much higher probably than the passive information gathering technique shown in black. From the second column, we see that our algorithm places significantly lower probability on attentiveness, which is correct because those users were distracted users. These are in line with the statistical analysis, with active information gathering doing a better job estimating the true human internal state. Fig.  5  plots the robot trajectories for the active information gathering setting. Similar to Fig.  4 , the solid lines are the mean of robot trajectories and the shaded regions show the standard error. We plot a representative dimension of the robot trajectory (like position or speed) for attentive (dark orange) or distracted (light orange) cases. The active robot probed the user, but ended up taking different actions when the user was attentive vs. distracted in order to maintain safety. For example, in Scenario 1, the trajectories show the robot is nudging into the human's lane, but the robot decides to move back to its own lane when the human drivers are distracted (light orange) in order to stay safe. In Scenario 2, the robot brakes in front of the human, but it brakes less when the human is distracted. Finally, in Scenario 3, the robot inches forward, but again it stops when if the human is distracted, and even backs up to make space for her. Fig.  6  plots the user trajectories for both active information gathering (first row) and passive information gathering (second row) conditions. We compare the reactions of distracted (light shades) and attentive (dark shades) users. There are large differences directly observable, with user reactions tending to indeed cluster according to their internal state. These differences are much smaller in the passive case (second row, where distracted is light gray and attentive is black). For example, in Scenario 1 and 2, the attentive users (dark orange) keep a larger distance to the car that nudges in front of them or brakes in front of them, while the distracted drivers (light orange) tend to keep a smaller distance. In Scenario 3, the attentive drivers tend to slow down and do not cross the intersection, when the robot actively inches forward. None of these behaviors can be detected clearly in the passive information gathering case (second row). This is the core advantage of active information gathering: the actions are purposefully selected by the robot such that users would behave drastically differently depending on their internal state, clarifying to the robot what this state actually is. Overall, these results support our simulation findings, that our algorithm performs better at estimating the true human internal state by leveraging purposeful information gathering actions.
 V. discussion Summary. In this paper, we formalized the problem of active information gathering between robot and human agents, where the robot plans to actively explore and gather information about the human's internal state by leveraging the effects of its actions on the human actions. The generated strategy for the robot actively probes the human by taking actions that impact the human's action in such a way that they reveal her internal state. The robot generates strategies for interaction that we would normally need to hand-craft, like inching forward at a 4-way stop. We evaluated our method in simulation and through a user study for various autonomous driving scenarios. Our results suggest that robots are indeed able to construct a more accurate belief over the human's driving style with active exploration than with passive estimation. Limitations and Future Work. Our work is limited in many ways. First, state estimation is not the end goal, and finding how to trade off exploration and exploitation is still a challenge. Second, our optimization is close to real-time, but higher computational efficiency is still needed. Further, our work relies on a model (reward function) of the human for each ϕ, which might be difficult to acquire, and might not be accurate. Thus far, we have assumed a static ϕ, but in reality ϕ might change over time (e.g. the human adapts her preferences), or might even be influenced by the robot (e.g. a defensive driver becomes more aggressive when the robot probes her). We also have not tested the users' acceptance of information gathering actions. Although these actions are useful, people might not always react positively to being probed. Last but not least, exploring safely will be of crucial importance.
 Conclusion. We are encouraged by the fact that robots can generate useful behavior for interaction autonomously, and are excited to explore informationgathering actions on human state further, including beyond autonomous driving scenarios. Passive Robot stops to R through Fig.  6 : The user trajectories for each scenario. The gap between attentive and distracted drivers' actions is clear in the active information gathering case (first row). 1 :Fig. 2 : 12 Fig.2: Our three scenarios, along with a comparison of robot plans for passive estimation (gray) vs active information gathering (orange). In the active condition, the robot is purposefully nudging in or braking to test human driver attentiveness. The color of the autonomous car in the initial state is yellow, but changes to either gray or orange in cases of passive and active information gathering respectively. driver might not realize the autonomous actions and maintain their velocity, getting closer to the autonomous vehicle. It is this difference in reactions that enables the robot to better estimate ϕ. Scenario 2: Braking to Explore on a Highway. In the second scenario, we show the driving style can be explored by the autonomous car probing the human driver behind it. The two vehicles start in the same lane as shown in Fig.2(b), where the autonomous car is in the front. In the passive condition, the autonomous car drives straight without exploring or enforcing any interactions with the human driven vehicle. In the active condition, the robot slows down to actively probe the human and find out her driving style. An attentive human would slow down and avoid collisions while a distracted human will have a harder time to keep safe distance between the two cars. Scenario 3: Nudging In to Explore at an Intersection. In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.2(c). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 Fig. 3 : 3 Fig. 3: Legends indicating active/passive robots, attentive/distracted humans, and real user/ideal model used for Fig.4, Fig.??, Fig.5, and Fig.6.
 Fig. 4 : 4 Fig. 4:The probability that the robot assigns to attentive as a function of time, for the attentive (left) and distracted (right). Each plot compares the active algorithm to passive estimation, showing that active information gathering leads to more accurate state estimation, in simulation and with real users.
 Fig. 5 : 5 Fig.5: Robot trajectories for each scenario in the active information gathering condition. The robot acts differently when the human is attentive (dark orange) vs. when the human is distracted (light orange) due to the trade-off with safety.
			 One exception is Nikolaidis et al. [17] , who propose to solve the full POMDP, albeit for discrete and not continuous state and action spaces.
			 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:30:44 UTC from IEEE Xplore. Restrictions apply.


Abstract:
The international governance of artificial intelligence (AI) is at a crossroads: should it remain fragmented or be centralised? We draw on the history of environment, trade, and security regimes to identify advantages and disadvantages in centralising AI governance. Some considerations, such as efficiency and political power, speak for centralisation. The risk of creating a slow and brittle institution, and the difficulty of pairing deep rules with adequate participation, speak against it. Other considerations depend on the specific design. A centralised body may be able to deter forum shopping and ensure policy coordination. However, forum shopping can be beneficial, and fragmented institutions could self-organise. In sum, these trade-offs should inform development of the AI governance architecture, which is only now emerging. We apply the tradeoffs to the case of the potential development of high-level machine intelligence. We conclude with two recommendations. First, the outcome will depend on the exact design of a central institution. A well-designed centralised regime covering a set of coherent issues could be beneficial. But locking-in an inadequate structure may pose a fate worse than fragmentation. Second, fragmentation will likely persist for now. The developing landscape should be monitored to see if it is self-organising or simply inadequate.
 Policy Implications • Secretariats of emerging AI initiatives, for example, the OECD AI Policy Observatory, Global Partnership on AI, the UN High-level Panel on Digital Cooperation, and the UN System Chief Executives Board (CEB) should coordinate to halt and reduce further regime fragmentation. • There is an important role for academia to play in providing objective monitoring and assessment of the emerging AI regime complex to assess its conflict, coordination, and catalysts to address governance gaps without vested interests. Secretariats of emerging AI initiatives should be similarly empowered to monitor the emerging regime. The CEB appears particularly well placed and mandated to address this challenge, but other options exist. • What AI issues and applications need to be tackled in tandem is an open question on which the centralisation debate sensitively turns. We encourage scholars across AI issues from privacy to military applications to organise venues to more closely consider this vital question. • Non-state actors, especially those with technical expertise, will have a potent influence in either a fragmented or centralised regime. These contributions need to be used, but there also need to be safeguards in place against regulatory capture. • The AI regime complex is at an embryonic stage, where informed interventions may be expected to have an outsized impact. The effect of academics as norm entrepreneurs should not be underestimated at this point.

Fragmentation and the Future: Investigating Architectures for International AI Governance

AI has the potential to dramatically alter the world for good or ill. These high stakes have driven a recent flurry of international AI policy making at the OECD, G7, G20, and multiple UN institutions. Scholarship has not kept pace with diplomacy. AI governance research to date has predominantly focused on national and sub-national levels  (Calo, 2017) . AI global governance research remains relatively nascent, focusing mostly on the proliferation of AI ethics principles  (Jobin et al., 2019)  and stocktaking of ongoing initiatives  (Garcia, 2020; Schiff et al., 2020) .  Kemp et al. (2019)  have called for specialised, centralised intergovernmental agencies to coordinate policy responses globally. Others have called for a centralised 'International Artificial Intelligence Organisation'  (Erdelyi and Goldsmith, 2018)  or an international coordinating mechanism under the G20  (Jelinek et al., 2020) . Conversely, some scholars favour more decentralised arrangements based around soft law, global standards, or existing international law instruments or UN multilateral organisations  (Cihon, 2019; Garcia, 2020; Kunz and O h Eigeartaigh, 2020; Wallach and Marchant, 2018) . This paper takes the initial step of considering the question: Should AI governance be centralised? The form of an international regime  1.  will fundamentally impact its operation and effectiveness. This includes the critical question of how an institutional form 'fits' the underlying problem  (Ekstrom and Crona, 2017; Young, 2002) . Questions of regime centralisation have occupied scholars and international negotiations for decades. The US diplomat George  Kennan (1970)  proposed the establishment of an 'International Environmental Agency' as an initial step towards an International Environmental Authority. The vexing question of whether to have a centralised body for environmental governance continued 42 years later during the Rio + 20 negotiations. There remains significant debate as to how much form affects performance and what level of centralisation is preferable, but there is little doubt that it is an important consideration for international regimes  (Biermann and Kim, 2020) . Centralisation is also a neglected area of examination for AI governance. The debate over form is in its infancy for AI with a few proposals for centralised regimes in academic literature and submissions to international processes  (Jelinek et al., 2020; Kemp et al., 2019) . Yet it seems unlikely that AI will be immune to increasing discussions and eventual political pushes for regime centralisation. Future negotiations over the form of AI governance will benefit immensely from early analysis. 'Centralisation', in this case, refers to the degree to which the coordination, oversight and/or regulation of a set of AI policy issues or technologies are housed under a single institution. Centralisation is relevant for policy makers and academics alike. A recent report by the UN Secretary General lamented the lack of coordination and inclusion among AI-related initiatives (United Nations  Secretary-General, 2020) . Early research and anticipatory initiatives may sensitively influence the path governance takes  (Stilgoe et al., 2013) . Scholars have a unique opportunity to be norm entrepreneurs and shape the emerging institutions through proactive, rather than retrospective, work on AI governance. The importance of this proactive approach has been emphasised for emerging technologies more broadly  (Rayfuse, 2017) . Moreover, choices made today may have long-lasting impacts as AI development continues  (Cave and O h Eigeartaigh, 2019) . In this paper, we explore the advantages and disadvantages of centralisation for AI governance. The defining problems of AI governance are threefold. The first is the political economy challenge and the importance of non-state actors' expertise in AI. The second is the need for anticipatory governance and technological foresight. The third is the variety and range of different AI applications, technologies, and policy problems. Our analysis hinges on a comparison with international regimes in three other domain areas, which display these core challenges, specifically environment, trade, and security. These three governance domains, while certainly distinct in important ways, are also arguably similar to AI governance across these dimensions: environmental governance invokes complex scientific questions that require technical expertise, has a broad scope encompassing transboundary and transsector effects, and includes a need for anticipation of future trends and impacts. Trade regimes span across a breadth of individual industries, and involve questions of standard-setting. Security and arms control regimes confront high-stakes situations and strategic interests, and a recurring need to 'modernise' regimes to track ongoing technological change. All three governance domains face questions of institutional inequalities. Finally, these regimes have been the subject of a rich literature exploring fragmentation and centralisations. We first outline the international governance challenges of AI, and review early proposed responses. We then draw on the conceptual frameworks of 'regime fragmentation'  (Biermann et al., 2009)  and 'regime complexes'  (G omez-Mera et al., 2020; Orsini et al., 2013) , and their application to the history of other international regimes, to identify considerations in designing a centralised regime complex for AI. We conclude with two practical recommendations.
 The state of AI governance Whether AI is a single policy area is actively debated. Some claim that AI cannot be cohesively regulated as it is a collection of disparate technologies, with different applications and risk profiles  (Stone et al., 2016) . This is an important but not entirely convincing objection. The technical field has no settled definition for 'AI',  2.  thus it is unsurprising that delineating a manageable scope for AI governance is difficult  (Schuett, 2019) . Yet this challenge is not unique to AI: definitional issues abound in areas such as environment and energy, but have not figured prominently in debates over centralisation. Indeed, energy and environment ministries are common at the domestic level. There are numerous ways in which a centralised body could be designed for AI governance. For example, a centralised approach could carve out a subset of interlinked AI issues. This could involve focusing on the potentially high-risk applications of AI systems, such as AI-enabled cyberwarfare, the use of natural language processing for information warfare, lethal autonomous weapons systems (LAWS), or high-level machine intelligence (HLMI).  3.  Another approach could govern underlying resource inputs for AI such as largescale compute hardware, software libraries, training datasets, or human talent. We are agnostic on the specifics of how centralisation could or should be implemented. We instead focus on the costs and benefits of centralisation in the abstract. The exact advantages and disadvantages of centralisation will vary with institutional design. Numerous AI issues could benefit from international cooperation. These include the high-risk applications mentioned above. It also encompasses more quotidian uses, such as AIenabled cybercrime; human health applications; safety and regulation of autonomous vehicles and drones; surveillance, privacy and data-use; and labour automation. This is not an exhaustive list of international AI policy issues. Global regulation across these issues is currently nascent, fragmented and evolving. OECD members and several other states agreed to a series of AI Principles, which were subsequently adopted by the G20 (OECD, 2020a). The Global Partnership on AI (GPAI) was launched by the G7 and several other states  (GPAI, 2020) . The fragmented membership in these initiatives is shown in Figure  1 . A wide range of UN institutions have begun to undertake some activities on AI  (ITU, 2019) . These developments are complimented by various treaty amendments, such as incorporating autonomous vehicles into the 1968 Vienna Convention on Road Traffic  (Kunz and O h Eigeartaigh, 2020)  or ongoing negotiations under the Convention on Certain Conventional Weapons (CCW) on LAWS. Private fora may also influence international governance (See  Green and Auld, 2017) , including the Partnership on AI and IEEE's Ethically Aligned Design initiative. The UN Secretary General intends to establish a multistakeholder advisory body on global AI cooperation (United Nations Secretary-General, 2020). UNESCO, the Council of Europe, and the OECD have similarly convened multistakeholder groups tasked with drafting policy instruments (Council of Europe (COE), 2020;  UNESCO, 2020; ; OECD, 2020b) . Whether these initiatives bear fruit, however, remains unclear, as many of the involved international organisations have fragmented membership, were not originally created to address AI issues and lack effective enforcement or compliance mechanisms (see  Morin et al., 2019) . For instance, while the US has endorsed the OECD AI Principles and while it eventually acquiesced to the GPAI, it has remained sceptical of hard, global rules . China, another global frontrunner in AI, is not a member of either body.  4.  How we initially structure international governance can be critical to its long-term success. Fragmentation and centralisation exist across a spectrum. Some fragmentation will always prevail, absent a global government. But the degree to which it prevails is crucial. Our definitions, including for fragmentation and key terms are provided in Table  1 . These definitions are by nature normatively loaded. For example, some may find 'decentralisation' to be a positive framing, while others may see 'fragmentation' to possess negative connotations. Recognising this, we use these terms in an analytical manner.
 Centralisation criteria: a history of governance trade-offs We explore a series of considerations for AI governance based on a review of existing scholarship on fragmentation  (Biermann and Kim, 2020; Biermann et al., 2009; Ostrom, 2010; Zelli and Asselt, 2013) . Specifically, political power and efficient participation support centralisation. The breadth vs. depth dilemma, as well as slowness and brittleness support decentralisation. Policy coordination and forum shopping considerations can cut both ways. This list is substantive, not exhaustive, and we intend it to open a discussion of design considerations for the nascent AI regime complex. It is far from the final word. Within each consideration below, we offer definitions, relevant regime histories, and discussion of implications for AI.
 Political power Regimes embody power in their authority over rules, norms, and knowledge beyond states' exclusive control. A more centralised regime sees this power concentrated among fewer institutions. A centralised, powerful architecture is likely to be more influential against competing international organisations and with constituent states  (Orsini et al., 2013) . Most environmental multilateral treaties, as well as UNEP, have faced sustained criticism for being unable to enact strong, effective rules or enforce them. In contrast, the umbrella of the WTO, has strongly enforced norms such as the most-favoured-nation principle (equally treating all WTO member states) have become the bedrock of international trade. Even to the extent of changing the actions of the US due to WTO rulings. The power and trackrecord of the WTO is so formidable that it has created a chilling effect: the fear of colliding with WTO norms and rules has led environmental treaties to actively avoid discussing or deploying traderelated measures  (Eckersley, 2004) . The power of this centralised body has stretched beyond the domain of trade to mould related issues. This is an area of high salience for AI. The creators and chief users of AI are 'big tech' companies which are some of the largest firms in the world by market capitalisation and have already had an enormous effect in shaping government policy  (Nemitz, 2018)  in favour of 'surveillance capitalism'  (Zuboff, 2019) . This daunting political economy challenge is perhaps the defining characteristic of AI. It seems unlikely that powerful vested economic and military interests in AI will be steered by a plethora of small bodies better than a single, well-resourced and empowered institution. Political power offers further benefits in governing emerging technologies that are inherently uncertain in both substance and impact. Uncertainty in technology and preferences has been associated with some increased centralisation in regimes  (Koremenos et al., 2001a) . There may also be benefits to housing a foresight capacity within the regime complex, to allow for accelerated or even proactive efforts  (Pauwels, 2019) , which would be particularly effective if centralised.
 Supporting efficiency and participation Decentralised AI governance may undermine efficiency and inhibit participation. States often create centralised regimes to reduce costs, for instance by eliminating duplicate efforts, yielding economies of scale within secretariats, and simplifying participation  (Esty and Ivanova, 2002) . Conversely, fragmented regimes may force states to spread resources and funding over many distinct institutions, limiting the ability of less well-resourced parties to participate  (Morin et al., 2019) . Historically, decentralised regimes have presented cost and participation concerns. Hundreds of related and sometimes overlapping international environmental agreements can create 'treaty congestion'  (Anton, 2012) . This complicates participation and implementation for both developed and developing nations  (Esty and Ivanova, 2002) . This includes costs associated with travel to different forums, monitoring and reporting for a range of different bodies, and duplication of effort by different secretariats  (Esty and Ivanova, 2002) . Similar challenges confront decentralised export regimes, which have notable duplication of efforts  (Brockmann, 2019) . These challenges are already evident in AI governance. Developing countries are not well represented at most international AI meetings (United Nations Secretary-General, 2020). Simultaneous and globally distributed meetings pose burdensome participation costs. Fragmented organisations must duplicatively invest in high-demand machine learning subject-matter experts to inform their activities. Centralisation would support institutional efficiency and participation.  Fragmentation or decentralisation A patchwork of international institutions which focus on a particular issue area but differ in scope, membership and often rules  (Biermann et al., 2009) .
 Centralisation The degree to which governance for an issue lies under the authority of a single body.
 Regime complex A network of three or more international regimes on a common issue area. These should have overlapping membership and cause potentially problematic interactions  (Orsini et al., 2013) . The costs and participation challenges posed by decentralisation may pose particular barriers to non-state actors  (Drezner, 2009) . AI-related expertise is primarily located in non-state actors today, namely multinational corporations and universities. Thus, barriers to non-state-actor participation in AI governance will pose particularly acute problems for writing rules that reflect the nature and development trajectory of AI technologies. However, these barriers may not limit all non-state actors from engaging in multiple fora. Indeed, those with sufficient resources may be able to pursue strategies to their advantage  (Kuyper, 2014) .
 Slowness and brittleness of centralised regimes One problem of centralisation lies in the relatively slow process of establishing centralised institutions, which may often be outpaced by the rate of (technological) change. Another challenge lies in centralised institutions' brittleness after they are established, that is, their vulnerability to regulatory capture or failure to react to changes in the issue area or technology. These issues are well reflected in challenges encountered in arms control regimes. Establishing new international institutions is often a slow process, especially with higher participation and stakes. Under the General Agreement on Tariffs and Trade (GATT), negotiations for a 26 per cent cut in tariffs between 19 countries took 8 months in 1947. The Uruguay round, beginning in 1986, took 91 months to achieve a tariff reduction of 38 per cent between 125 parties  (Martin and Messerlin, 2007) . Historically, international law has been quicker at responding to technological change than to other changes; but even there its record is chequered, in some cases (e.g., spaceflight) adjusting within years, while being far more delayed in others (e.g., modern anti-personnel landmines)  (Picker, 2001) . Decentralised efforts might prove quicker to respond, especially if they rely more on informal institutions with a smaller, like-minded membership  (Morin et al., 2019) . Centralised governance may be particularly vulnerable to lengthy negotiations, especially if a few states hold unequal stakes in a technology, or if there are significant differences in information and expertise among state and private actors  (Picker, 2001) . AI fulfils both of these conditions. Moreover, because AI technology develops rapidly, slow implementation of rules and principles could enable certain actors to take advantage by setting de facto rules. Even after its creation, a centralised regime can be brittle. The very qualities that provide it with political power may exacerbate the adverse effects of regulatory capture, and features that ensure institutional stability may also lead to an inability to adapt to new conditions. The regime might break before it bends. The first potential risk is regulatory capture. As illustrated by numerous cases, including undue corporate influence in the World Health Organisation during the 2009 H1N1 pandemic  (Deshman, 2011) , no institution is fully immune to capture, and centralisation may facilitate this by providing a single locus of influence  (Martens, 2017) . On the other hand, a regime complex comprising many smaller, parallel institutions could find itself vulnerable to capture by powerful actors, who can afford representation in every forum. Some have already expressed concern about the resources and sway of private tech actors in AI governance  (Nemitz, 2018) , and proposals for AI governance have been surrounded by calls to ensure their independence from such influence (Nature Editors, 2019). Moreover, centralised regimes entail higher stakes. International institutions can be notoriously path-dependent and fail to adjust to changing circumstances  (Baccaro and Mele, 2012) . The public failure of a flagship global AI institution could have lasting political repercussions. It could strangle subsequent proposals in the crib, by undermining confidence in multilateral governance generally or on AI issues specifically. By contrast, for a decentralised regime complex to similarly fail, all of its component institutions would need to 'break' or fail to innovate simultaneously. A centralised institution that does not outright collapse, but which remains ineffective, may inhibit better efforts. Ultimately, brittleness is not an inherent weakness of centralisation, but rather may depend on institutional design. There may be strategies to 'innovation-proof'  (Maas, 2019a)  governance regimes. Periodic renegotiation, modular expansion, additional protocols to framework conventions, 'principles based regulation', or sunset clauses can also support ongoing adaptation (see  Marchant et al., 2011) . This discussion intersects with debates over whether a new centralised regime is even possible in today's shifting, dense institutional landscape  (Alter and Raustiala, 2018; Morin et al., 2019) . The speed of capability development in AI also highlights questions over the relative 'speed' or 'responsiveness' of different regime configurations. In slowmoving areas, a centralised regime's slowness may not be a problem. However, technological change has often 'perforated' many arms control regimes, from the Nuclear Non-Proliferation Treaty to the Missile Technology Control Regime, which sometimes struggled to carry out muchneeded 'modernisation' in provisions or export control lists  (Nelson, 2019) . This raises questions of necessary institutional speed. Is AI an issue that is so fast it makes centralisation untenable, such that we need a decentralised regime to match its speed and complexity? Or, should we use a singular institutional anchor to slow and channel the technology's development or application? There is precedent for international instruments directing or curtailing the development of certain technologies. The 1978 Environmental Modification Convention (ENMOD) Convention was an effective tool in preventing both funding for geoengineering research and the weaponised deployment of weather manipulation. By 1979, US investments in such technologies had dramatically decreased  (Fleming, 2006) .
 The breadth vs. depth dilemma Pursuing centralisation may create an overly high threshold that limits participation. Many multilateral agreements face a trade-off between having higher participation ('breadth') or stricter rules and greater ambition of commitments ('depth'). The dilemma is particularly evident for centralised institutions that are intended to be powerful and require strong commitments from states. Sacrificing depth for breadth can also pose risks. The 2015 Paris Agreement on Climate Change was watered down to allow for the legal participation of the US. Anticipated difficulties in ratification through the Senate led to negotiators opting for a 'pledge and review' structure with few legal obligations, which permitted the US to join through executive approval  (Kemp, 2017) . In this case, inclusion of the USwhich proved temporarycame at the cost of cutbacks to the demands which the regime made on all parties. In contrast, decentralisation could allow for major powers to engage in at least some regulatory efforts, where they would be deterred from signing up to a more comprehensive package. This has precedence in climate governance. Some claim that the US-led Asia-Pacific Partnership on Clean Development and Climate helped, rather than hindered climate governance, as it bypassed the UN Framework Convention on Climate Change (UNFCCC) deadlock and secured (non-binding) commitments from actors not bound by the Kyoto Protocol  (Zelli, 2011) . This matters, as buy-in may prove a particular thorny issue for AI governance. The actors who lead in AI development include powerful states, such as the US and China, that are potentially most adverse to restrictive global rules. They have thus far proved unenthusiastic regarding the global governance of security issues such as anti-personnel mines, LAWS, and cyberwarfare. In response, governance could take a different approach to military uses of AI. Rather than seeking a comprehensive agreement, devolving and spinning off certain components into separate treaties (e.g., separately covering LAWS testing standards; measures for liability and responsibility; or limits to operational context) could instead allow for the powerful to ratify and move forward some of those options  (Weaver, 2014) . The breadth vs. depth dilemma is a trade-off in multilateralism generally, and a key challenge for centralisation. The benefit of a centralised body would be to create a powerful anchor that ensures policy coordination and coherence. In many cases, it will likely need to restrict membership to have teeth, or lose its teeth to secure wide participation. For specific issues in AI governance, this 'breadth vs. depth' trade-off might inform relative expectations of ongoing AI governance initiatives. If 'breadth' is more important, one might put more stock in nascent efforts at the UN  (Garcia, 2020) ; if 'depth' of commitment seems more important, one might instead favour initiatives of like-minded states such as the GPAI. The evolving architecture of AI governance suggests that a 'critical mass governance'  (Kemp, 2017)  approach may be appropriate. That is, there is a single centralised, framework under which progressive clubs move forward on particular issues. Rather than having an array of treaties, one has a set of protocols for different technologies or applications under a single framework. A similar approach has been taken in treaties such as the 1983 Convention on Long-Range Transboundary Air Pollution.
 Forum shopping Forum shopping may help or hinder AI governance. Fragmentation enables actors to choose where and how to engage. Such 'forum shopping' may take one of several forms: shifting venues, abandoning one, creating new venues, and working to sew competition among multiple  (Braithwaite and Drahos, 2000) . Even when there is a natural venue for an issue, actors have reasons to forum shop. For instance, states may look to maximise their influence  (Pekkanen et al., 2007) , and placate constituents by shifting to a toothless forum  (Helfer, 2004) . Membership in AI initiatives is highly varied and as initiatives begin to consider binding instruments, this ranging membership may be exploited. The ability to successfully forum-shop depends on an actor's power. Most successful examples of forum-shifting have been led by the US  (Braithwaite and Drahos, 2000) . Intellectual property rights (IPR) in trade, for example, were subject to prolonged, contentious forum shopping. Developed states resisted attempts of the UN Conference on Trade and Development (UNCTAD) to address the issue by trying to shift it onto the World Intellectual Property Organisation (WIPO)  (Braithwaite and Drahos, 2000)  and then subsequently to the WTO  (Helfer, 2004) , despite protests from developing states. But weak states and non-state actors can also pursue forum shopping strategies in order to challenge the status quo, sometimes with success  (Jupille et al., 2013) . For example, developing states further shifted some IPR in trade to the WHO, and subsequently won concessions at the WTO  (Kuyper, 2014) . Forum shopping may help or hurt governance  (G omez-Mera, 2016) . This is evident in current efforts to regulate LAWS. While the Group of Governmental Experts has made some progress, on the whole the CCW has been slow. In response, activists have threatened to shift to another forum, as happened with the Ottawa Treaty that banned anti-personnel mines  (Delcker, 2019) . This strategy could catalyse progress, but also brings risks of further forum shopping. Forum shopping may similarly delay, stall, or weaken regulation of time-sensitive AI policy issues, including potential HLMI development. Non-state actors that participate in multiple fora may influence regime complex evolution, though perhaps to the detriment of other weak actors  (Orsini, 2013) . Thus, leading AI firms likely have sway when they elect to participate in some venues but not others. To date, leading AI firms appear to be prioritising engagement at the OECD over the UN. A decentralised regime will enable forum shopping, though further work is needed to determine whether this will help or hurt governance outcomes.
 Policy coordination There are good reasons to believe that either centralisation or fragmentation could enhance coordination. A centralised regime can enable easier coordination both across and within policy issues, acting as a focal point for states. Alternatively, fragmented institutions may be mutually supportive and even more creative. Centralisation reduces the incidence of conflicting mandates and enables communication. These are the ingredients for policy coherence, as shown in the case of the WTO above under 'political power'. However, fragmented regimes can often act as complex adaptive systems. Political requests and communication between secretariats can ensure bottom-up coordination. Multiple organisations have sought to reduce greenhouse gas emissions within their respective remits, often at the behest of the UNFCCC Conference of Parties. Sometimes effective, bottom-up coordination can slowly evolve into centralisation. Indeed, this was the case for the GATT and numerous regional, bilateral and sectoral trade treaties, which all coalesced together into the WTO. While this organic self-organisation has occurred, it has taken decades. Some have argued that 'polycentric' governance approaches may be more creative and legitimate than centrally coordinated regimes  (Acharya, 2016; Ostrom, 2010) . Arguments in favour of polycentricity include the notion that it enables governance initiatives to begin having impacts at diverse scales, and that it enables experimentation with policies and approaches  (Ostrom, 2010) . Consequently, these scholars assume 'that the invisible hand of a market of institutions leads to a better distribution of functions and effects'  (Zelli and van Asselt, 2013, p. 7 ). Yet an absence of centralised authority to manage regime complexes has presented challenges in the past. Across the proliferation of Multilateral Environmental Agreements (MEAs) there is no requirement to cede responsibility to the UN Environmental Programme in the case of overlap or competition. This has led to turf wars, inefficiencies and even contradictory policies  (Biermann et al., 2009) . One of the most notable examples is that of hydrofluorocarbons (HFCs). HFCs are potent greenhouse gases, and yet their use was encouraged by the Montreal Protocol since 1987 as a replacement for ozone-depleting substances. This was only resolved via the 2016 Kigali Amendment to the Protocol. It is unclear if the different bodies covering AI issues will self-organise or collide. Many of the issues are interdependent and need to be addressed in tandem. Some policylevers, such as regulating computing power or data, will impact multiple areas, given that AI development and use is closely tied to such inputs. Numerous initiatives on AI and robotics are displaying loose coordination  (Kunz and O h Eigeartaigh, 2020) . But it remains uncertain whether the virtues of a free market of governance will prevail. Great powers can exercise monopsony-like influence through forum shopping, and the supply of both computing power and machine learning expertise are highly concentrated. In sum, centralisation can reduce competition and enhance coordination, but it may suffocate the creative self-organisation of decentralised arrangements. Discussion: what would history suggest?
 Summary of considerations The multilateral track record and peculiarities of AI yield suggestions and warnings for the future. A centralised regime could lower costs, support participation, and act as a powerful new linchpin within the international system. Yet centralisation could simply produce a brittle dinosaur, of symbolic value but with little meaningful impact. A poorly executed attempt at centralisation could lock-in a fate worse than fragmentation. Policy making and research alike could benefit from addressing the considerations presented in this paper, a summary of which is presented in Table  2 .
 The limitations of 'centralisation vs. decentralisation' debates Structure is not a panacea. Specific provisions such as agendas and decision-making procedures matter greatly, as do the surrounding politics. Underlying political will may be impacted by framing or connecting policy issues  (Koremenos et al., 2001b) . The success of a regime depends on design details. Moreover, institutions can be dynamic, and broaden over time by taking in new members or deepen in strengthening commitments. Successful multilateral efforts, such as trade and ozone depletion, tend to do both. Yet, decisions taken early on constrain and partially determine future paths. This dependency can even take place across regimes. The Kyoto Protocol was largely shaped by the targets-and-timetables approach of the Montreal Protocol, which itself drew from the Convention on Long-range Transboundary Air Pollution. This targets-and-timetables approach continues today in the way that most countries frame their climate pledges to the Paris Agreement. The choices we make on governing shortterm AI challenges will likely shape the management of other policy issues in the long term  (Cave and O h Eigeartaigh, 2019 ). Yet, committing to centralisation, even if successful, may not solve the right problemwhich may be geopolitical, not architectural. Centralisation could even exacerbate the problem by diluting scarce political attention, incurring heavy transaction costs, and shifting discussions away from bodies which have accumulated experience  (Juma, 2000) . For example, the Bretton Woods Institutions of the IMF and World Bank, joined later by the WTO, are centralised regimes that engender power. However, those institutions had the express support of the US and may have simply manifested state power in institutional form. Efforts to ban LAWS and create a cyberwarfare convention have been broadly opposed by states with an established technological superiority in these areas  (Eilstrup-Sangiovanni, 2018) .
 HLMI: An illustrative example The promise of centralisation may differ by policy issue. HLMI is one issue that is markedly unique: it is distinct in its risk profile, uncertainty, and linkage to other AI policy issues. While timelines are uncertain, the creation of such advanced AI systems is the express goal of various present-day projects  (Baum, 2017) , and the future development of an 'unaligned' HLMI could have catastrophic consequences  (GCF, 2018) . The creation of HLMI could lead to grotesque power imbalances. It could also exacerbate other AI policy problems, such as labour automation and advanced military applications. In Table  3  we provide a brief application of our framework to HLMI. It shows that centralisation of governance is particularly promising for HLMI. This is due to its neglect, stakes, scope, and need for informed, anticipatory policy. Rather than any AI governance blueprint, our trade-offs framework provides one way of thinking through the costs and benefits of centralising governance. Identifying areas which are more easily defined and garner the benefits of centralised regulation provides an organic approach to thinking through which subset of topics an AI umbrella body could cover.
 Lessons for theory This is the first application of regime complex theory to the problem of AI governance. It is timely and pertinent given the nascent state of AI governance and of the technology itself. While the majority of the literature has observed mature regimes retrospectively, AI offers an opportunity for scholars to both track and influence the development of a new regime complex from its earliest stages. Our analysis highlights both the uses and limits of the theoretical regime complex lens for AI. It can elucidate many important trade-offs, but provides little help in navigating the underlying geopolitics. The six considerations we have identified are also certainly not exhaustive of regime complex theory; further work could explore the complementary dynamics such as issue linkage, regime 'interplay management', or norm cascades in AI governance. Beyond this, the literature needs a better understanding of three key areas that are central to AI. First, what does the political economy of AI mean for AI governance and centralisation? Regulatory capture is a genuine threat, yet many non-state actors hold valuable technical knowledge. Some, such as machine learning developers and NGOs have been influential in shaping governance on lethal autonomous weapons  (Belfield, 2020) . How these actors can shape the choice of fora and influence states under centralisation or decentralisation is pivotal. Second, how should institutions match the speed of evolving collective action problems? Is the aim to make governance agile enough to keep pace with accelerating technological change or to manage the pace or direction of such changes to levels that are socially and politically manageable? Theoretically, foresight methodologies have rarely been considered in regime complex debates. Yet for fastmoving and high-stakes technologies, they should be. Theory will need to better address how foresight and development trajectory monitoring capabilities intersect with the debates over governance architecture. Third, how will these considerations look for particular institutional structures? We have presented a cursory case of HLMI and noted that there is an active debate of how to define AI and structure its governance. How will the case for centralisation look for a regime which targets just high-risk or military applications? Our framework provides an easily deployed way to analyse more discrete proposals for AI governance in the future.
 Lessons for policy Our framework provides a tool for policy makers to inform their decisions of whether to join, create, or forgo new AI policy institutions. For instance, the recent choice of whether to support the creation of an independent Global Panel on AI (GPAI) involved these considerations. Following the US veto at the G7 in 2019, GPAI was established in close relationship with the OECD. For now, it is worth monitoring the current landscape of AI governance to see if it exhibits enough policy coordination and political power to effectively deal with mounting AI policy problems. While there are promising initial signs  (Kunz and O h Eigeartaigh, 2020 ) there are also already impending governance failures, such as for LAWS and cyberwarfare. We outline a suggested monitoring method in Table  4 . There are three areas to monitor: conflict, coordination, and catalyst. Conflict should measure the extent to which principles, rules, regulations, and other outcomes from different bodies in the AI regime complex undermine or contradict each other. Coordination seeks to measure the proactive steps that AI-related regimes take to work with each other. This includes liaison relationships, joint initiatives, and reinforcement between outputs and principles. Catalyst raises the important question of governance gaps: is the regime complex self-organising to proactively address international AI policy problems? Numerous AI policy problems currently have no clear coverage under international law. Monitoring these regime complex developments, using various existing and emerging tools (see  Maas, 2019b; Deeks, 2020) , could inform a discussion and decision of whether to centralise AI governance further. The international governance of AI is nascent and fragmented. Centralisation under a well-designed, modular, 'innovation-proof', critical mass framework may be a desirable solution. However, such a move must be approached with caution. Defining its scope and mandate is one problem. Ensuring a politically-acceptable and well-designed body is perhaps a more daunting one. For now, we should closely watch the trajectory of both AI technology and its governance initiatives to determine whether centralisation is worth the risk. Figure 1 . 1 Figure 1. Membership in selected international AI policy initiatives.
 Table 1 . 1 Definition of key governance terms GPAI Austria OECD Principles Belgium Chile Costa Rica Ireland Colombia Israel New Zealand Czech Republic Latvia Slovenia Denmark Lithuania Singapore Estonia Finland Luxembourg Malta Greece The Netherlands Hungary Norway Australia Iceland Peru Canada Poland France Portugal Germany Romania Italy Slovakia Japan Spain Korea Sweden EU * India Mexico UK USA Argentina Brazil Turkey Switzerland Ukraine 142 UN member states are not China represented Indonesia Russia Saudi Arabia South Africa * as a member in G20 Principles its own right Term Definition
 Table 2 . 2 Summary of considerations Implications for Consideration centralisation Historical example AI policy issue example Political power Pro Shaping other regimes: WTO has created a Influencing powerful vested economic and chilling effect such that environmental treaties military interests in AI may require a single avoid trade-related measures. empowered institution. Efficiency & Pro Decentralisation raises inefficiencies and barriers: Fragmentation requires duplicative investment participation Proliferation of multilateral environmental in AI subject-matter experts and undermines agreements poses challenges in negotiation, participation from developing countries and implementation, and monitoring. non-state actors. Slowness & Con Slowness: Under the GATT, 1947 tariff Process of centralised regime development brittleness negotiations among 19 countries took may not keep pace with the speed of AI 8 months. The Uruguay round, beginning in development. 1986, took 91 months for 125 parties to agree on reductions. Regulatory capture: WHO accused of undue corporate influence in response to 2009 H1N1 pandemic. Breadth vs. depth Con Watering down: 2015 Paris Agreement suggests Attempts to effectively govern the military dilemma attempts to 'get all parties on board' may uses of AI have been resisted by the most require less-stringent rules. powerful states. Forum shopping Depends on Power predicts outcomes: Actors can use forum shopping to either design Developed countries shifted IPR in trade from undermine or catalyse progress on UNCTAD to WIPO to WTO. governance regimes for military AI systems. Accelerates progress: NGOs and some states shifted away from CCW to ban anti-personnel mines. Policy coordination Depends Strong, but delayed convergence: Numerous AI governance initiatives display on design GATT and numerous trade treaties coalesced into loose coordination, but it is unclear if these the WTO after decades initiatives can respond to developments in a Contradictory policies: timely manner. Montreal Protocol promoted the use of potent greenhouse gases for nearly thirty years.
 Table 3 . 3 An application of the framework to high-level machine intelligence (HLMI) If short HLMI timelines (less than 10-15 years) are expected, the lengthy period to negotiate and create such a body would be a critical weakness. If longer timelines are expected, there should be sufficient time to develop a centralised institution. Institutional capture is a concern given the resourced corporate actors involved in creating HLMI, e.g., Google or OpenAI. However, it is unclear if capture would be more likely under a centralised body. Policy coordination Policy coordination is key for HLMI. It has close connections to issues such as labour automation and automated cyberwarfare. The creation or use of HLMI is not directly regulated by any treaties or legal instruments. This makes the creation of a new, dedicated institution to address it easier and less unlikely to trigger turf wars. However, it also makes it less likely that the existing tapestry of global governance can self-organise to cover HLMI in a timely manner. Consideration HLMI Political power Potential catastrophic risks make the increased political power of a centralised institution desirable. The creation of HLMI is a potential 'free-driver' issue. An effective response needs to have the teeth to deter major players from acting unilaterally. This will require a coordinated effort to track and forecast HLMI project efforts (see Baum, 2017), as well as a politically empowered organisation to act upon this information. Efficiency & Centralisation would support economies of participation scale in expertise to support efficient governance. Given the significant resources and infrastructure likely needed, a joint global development effort could be an efficient way to govern HLMI research. Slowness & brittleness Depth vs. breadth Costs and requisite capabilities may restrict dilemma the development of HLMI to a few powerful players. Fewer actors makes centralisation more feasible. The breadth vs. depth dilemma could be avoided through a 'critical mass' approach that initially involves only the few countries that are capable of developing HLMI, although there would be legitimacy benefits to expanding membership. Forum shopping A centralised body is well placed to prevent forum shopping, as there is currently no coverage of HLMI development and deployment under international law. Future forum shopping could undermine timely negotiations amid risky HLMI development.
 Table 4 . 4 Regime complex monitoring suggestions Key theme Questions Methods Conflict To what extent are Expert and practitioner regimes' principles survey Network and outputs in analysis (e.g., citation opposition over time? network clustering Coordination Are regimes taking and centrality) steps to complement Natural Language each other? Processing (e.g., Catalyst Are regimes self- textual entailment and organizing to fact checking) proactively fill governance gaps?
			 © 2020 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.Global Policy (2020) 11:5
			 Global Policy (2020) 11:5 © 2020 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.Architectures for International AI Governance
			 Global Policy (2020) 11:5 © 2020 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Global Policy - 2020 - Cihon - Fragmentation and the Future Investigating Architectures for International AI Governance.tei.xml