Abstract: Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. To this end, prior work has developed teaching protocols that enable agents to learn efficiently in complex environments. In many of these methods, the teacher's guidance is tailored to agents with a particular representation or underlying learning scheme, offering effective but highly specialized teaching procedures. In this work, we introduce protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema, and conduct preliminary experiments evaluating the effectiveness of protocols in simple domains.

David Abel
John Salvatier
Andreas Stuhlmüller
Owain Evans

Agent-Agnostic Human-in-the-Loop Reinforcement Learning

Introduction A central goal of Reinforcement Learning (RL) is to design agents that learn in a fully autonomous way. An engineer designs a reward function, input/output channels, and a learning algorithm. Then, apart from debugging, the engineer need not intervene during the actual learning process. Yet fully autonomous learning is often infeasible due to the complexity of real-world learning problems, the difficulty of specifying reward functions that lead to good performance, and the presence of potentially dangerous outcomes. Consider a robot learning to perform household chores. Human engineers design and supervise a curriculum, moving the agent between simulation, practice environments, and real house environments. Over time, they might tweak reward functions, heuristics, and state representations. They may intervene directly in real-world training to prevent the robot damaging itself, destroying valuable goods, or harming people it interacts with. In this example, humans do not just design the learning agent: they are also "in the loop" of the agent's learning process. Many learning systems have a human in the loop. Self-driving cars learn with humans ready to intervene in dangerous situations. Facebook's algorithm for recommending "trending" news stories has humans filtering out inappropriate content  [1] . In both examples, the agent's environment is complex, non-stationary, and there are a wide range of damaging outcomes (like a traffic accident). As RL is applied to increasingly complex real-world problems, such interactive guidance will become more important.
 Our contribution: agent-agnostic guidance of RL algorithms Our goal is to develop a framework for human-agent interaction that is (a) agent-agnostic and (b) can capture a wide range of ways a human can help an RL agent. This paper targets environments where the state is fully observed; that is, the learning agent interacts with a Markov Decision Process (MDP)  [33, 20, 36] . Prior literature has investigated how people can help RL agents learn more efficiently through different methods of interaction  [29, 22, 25, 32, 37, 43, 19, 16, 23, 38, 21, 44, 40, 39, 27, 12] . Often, the human's role is to pass along knowledge about relevant quantities of the RL problem, like Q-values, action optimality, or the true reward for a particular state-action pair. This way, the person can bias exploration, prevent catastrophic outcomes, and accelerate learning. Most existing work develops agent-specific protocols for human interaction. That is, protocols for human interaction or advice that are designed for a specific RL algorithm (such as Q-learning). For instance, Griffith et al.  [16]  investigate the power of policy advice for a Bayesian Q-Learner. Other works assume that the states of the MDP take a particular representation, or that the action space is discrete or finite. Making explicit assumptions about the agent's learning process can enable more powerful teaching protocols that leverage insights about the learning algorithm or representation. Agent-specific protocols for human interaction can be contrasted with agent-agnostic protocols. An agent-agnostic protocol is a single protocol that can be plugged into any RL agent, such as an agent based on Q-learning, policy gradient, MCTS, Deep Q-learning, and so on, and that can provide advice/guidance to any such agent. There are obvious disadvantages to agent-agnostic protocols. The agent is not specialized to the protocol, so it is unable to use Active Learning (asking the human informative questions as in  [4] ) and will not automatically have an observation model that faithfully represents the process the human uses to generate advice (as in  [16, 19] ). Likewise, the human cannot provide optimally informative advice to the agent as they don't know the agent's prior knowledge, exploration technique, representation, or learning method. Conversely, agent-specific protocols may perform well for one type of algorithm or environment, but poorly on others. When researchers tackle challenging RL problems, they tend to explore a large space of algorithms with important structural differences: some are model-based vs. model-free, some approximate the optimal policy, others a value function, and so on. It is substantial effort to adapt a human-advice protocol to each such algorithm. Moreover, as advice protocols and learning algorithms become more complex, greater modularity will help limit design complexity. In our framework, the interaction between the person guiding the learning process, the agent, and the environment is formalized as a protocol program. This program controls the channels between the agent and the environment based on human input (see Figure  1 ). This allows the human extensive control over the agent: in an extreme case, the agent can be prevented from interacting with the real environment entirely and only interact with a simulation. At the same time, we require that the human only interact with the agent during learning through the protocol program-both agent and environment are a black box to the human.
 Framework Any system for RL with a human in the loop has to coordinate three components: The environment is an MDP and is specified by a tuple M = (S, A, T , R, γ), where S is the state space, A is the action space, T : S × A × S → [0, 1], denotes the transition function, a probability distribution on states given a state and action, R : S × A → R is the reward function, and γ is the discount factor. The agent is a (stateful, potentially stochastic) function L : S × R → A. The human can receive and send advice information of flexible type, say X in and X out , so, we will treat the human as a (stateful, potentially stochastic) function H : X in → X out . For example, X in might contain the history of actions, states, and rewards so far, and a new proposed action a , and X out might be an action as well, either equivalent to a (if accepted) or different (if rejected). We assume that the human knows in general terms how their responses will be used and is making a good-faith effort to be helpful. The interaction between the environment, the agent, and a human advisor sets up a mechanism design problem: how can we design an interface that orchestrates the interaction between these components such that the combined system maximizes the expected sum of γ-discounted rewards from the environment? In other words, how can we write a protocol program P : S × R → A that can take the place of a given agent L, but that achieves higher rewards by making efficient use of information gained through sub-calls to L and H? By formalizing existing and new techniques as explicit programs, we facilitate understanding and comparison of these techniques within a common framework. By abstracting from particular agents and environments, we may be able to design mechanisms that scale to more advanced agents  [7] . Algorithm 1 Agent in control (standard) 1: procedure AGENTCONTROL(s, r) 2: return L(s, r) 3: end procedure Algorithm 2 Human in control 1: procedure HUMANCONTROL(s, r) 2: return H(s, r) 3: end procedure Algorithm 3 Training in simulation 1: M * = (S, A, T * , R * , γ) Simulation 2: η = [] History: array of (S × R × A) 3: procedure TRAININSIMULATION(s, r) 4: s = s 5: r = r 6: while H(η) = "agent is ready" do r = H(s, r) 3: return L(s, r) 4: end procedure Figure  2 : Many schemes for human guidance of RL algorithms can be expressed as protocol programs. These programs have the same interface as the agent L, but can be safer or more efficient learners by making use of human advice H.
 Capturing Existing Advice Schemes Protocol programs cannot capture all advice protocols. Any protocol that depends on prior knowledge of the agent's learning algorithm, priors or hyper-parameters is ruled out. Despite this constraint, the framework can capture a range of existing protocols where a human-in-the-loop guides an agent. Figure  1  shows that the human can manipulate the actions sent to the environment and the agent's observed states and rewards. This points to a combinatorial set of protocols, including reward shaping, action pruning, providing hand-coded features of state, and training in simulation.
 Reward shaping Section 2 defined the reward function R as part of the MDP M . However, while humans don't design real-world transition functions, we do design reward functions. Usually the reward function is hand-coded prior to learning and must accurately assign reward values to any state the agent might reach. An alternative is to have a human generate the rewards "interactively". The human observes the state and action and returns a scalar to the agent. This setup has been explored in work on TAMER  [21] . A similar setup (with an agent-specific protocol) was applied to robotics by Daniel et al.  [8] . It is straightforward to represent rewards that are generated interactively (or "online") using protocol programs. We now turn to other protocols in which the human manipulates rewards. These protocols assume a fixed reward function R that is part of the MDP M .
 Reward shaping and Q-value initialization In Reward Shaping protocols, the human engineer changes the rewards given by some fixed reward function in order to influence an agent's learning. Ng et al.  [29]  introduced "potential-based" shaping, which shapes rewards without changing an MDP's optimal policy. In particular, each reward received by the environment is augmented by a shaping function: F (s, a, s ) = γφ(s ) − φ(s), (1) so the agent actually receives r = F (s, a, s ) + R(s, a). Wiewiora et al.  [43]  showed potential shaping to be equivalent (for Q-learners) to a subset Q-value initialization under some assumptions. Further, Devlin and Kudenko  [10]  propose dynamic potential shaping functions that change over time. That is, the shaping function F also takes as two time parameters, t and t , such that: F (s, t, s , t ) = γφ(s , t ) − φ(s, t) (2) Where t > t. Their main result is that dynamic shaping functions of this form also guarantee optimal policy invariance. Similarly, Wiewiora et al.  [43]  extend potential shaping to potential-based advice functions, which identifies a similar class of shaping functions on (s, a) pairs. In Section 4, we show that our Framework captures reward shaping, and consequently, some notion of Q-value initialization.
 Training in Simulation It is common practice to train an agent in simulation and transfer it to the real world once it performs well enough. Algorithm 3 (Figure  2 ) shows how to represent this as a protocol program. The idea is as follows: Let M be the real-world'. The protocol program includes a simulator M * . The agent L interacts with M * while the human observes performance and at some point transfers L to M . 1
 Action Pruning Action pruning is a technique for dynamically removing actions from the MDP to reduce the branching factor of the search space. Such techniques have been shown to accelerate learning and planning time  [35, 17, 34, 2] . In Section 5, we apply action-pruning to the problem of preventing catastrophic actions ("Safe RL"). Protocol programs allow action pruning to be carried out interactively. Instead of having to decide which actions to prune prior to learning, the human can wait to observe the states that are actually encountered by the agent, which may be valuable in cases where the human has limited knowledge of the environment or the agent's learning ability. In Section 4, we exhibit an agent-agnostic protocol for interactively pruning actions that preserves the optimal policy. Our pruning protocol is illustrated in a gridworld with lava pits (Figure  3 ). The agent is represented by a gray circle, "G" is a goal state that provides reward +1, and the red cells are lava pits with reward −200. All white cells provide reward 0. At each time step, the human checks whether the agent moves into a lava pit. If it does not (as in moving "DOWN" from state 34), the agent continues as normal. If it does (as in moving "RIGHT" from state 33), the human sends a "STAY" action to the MDP (preventing movement right) and a "next state" of 33 to the agent. The agent doesn't actually fall in the lava but the human sends them a reward r ≤ −200. After this negative reward, the agent is unlikely to try the action again. For the protocol program, see Algorithm 4 in Figure  2 . Note that the agent receives no explicit signal that their attempted catastrophic action was blocked by the human. They observe a big negative reward and a "self-loop" state transition but no information about whether the human or environment generated their observation.
 Manipulating state representation The agent's state representation can have a significant influence on its learning. Suppose the MDP M has a "raw" state vector s. The human engineer can specify a mapping φ such that the agent always receives φ(s) = s in place of s. Such mappings are used to specify high-level features of state that are important for learning, or to dynamically ignore confusing features from the agent. Other methods have focused on state abstraction functions of this form that can decrease learning time and preserve the quality of learned behavior, as in  [24, 30, 13, 18, 9, 3, 14] .
 Theory Here we illustrate some simple ways in which our proposed agent-agnostic interaction scheme captures other existing agent-agnostic protocols. The following results all concern Tabular MDPs, but are intended to offer intuition for high-dimensional or continuous environments as well.
 Reward Shaping First we observe that protocol programs can precisely capture methods for shaping reward functions. Remark 1: For any reward shaping function F , including potential-based shaping, potential-based advice, and dynamic potential-based advice, there is a protocol that produces the same rewards. To construct such a protocol for a given F , simply let the reward output by the protocol, r, take on the value F + r at each time step. That is, in Algorithm 5, simply define H(s, r) = F (s) + r.
 Action Pruning We now show that there is a simple class of protocol programs that carry out action pruning of a certain form. Remark 2: There is a protocol for pruning actions in the following sense: for any set of state action pairs sa ⊂ S × A, the protocol ensures that, for each pair (s i , a j ) ∈ sa, action a j is never executed in the MDP in state s i . The protocol is as described in Section 3.3 and shown in Algorithm 4. The premise is this: in all cases where the agent executes an action that should be pruned, the protocol gives the agent low reward and forces the agent to self-loop. Knowing which actions to prune is itself a challenging problem. Often, it is natural to assume that the human guiding the learner knows something about the environment of interest (such as where high rewards or catastrophes lie), but may not know every detail of the problem. Thus, we consider a case in which the human has partial (but useful) knowledge about the problem of interest, represented as an approximate Q-function. The next remark shows there is a protocol based on approximate knowledge with two properties: (1) it never prunes an optimal action, (2) it limits the magnitude of the agent's worst mistake: Remark 3: Assuming the protocol designer has a β-optimal Q function: ||Q * (s, a) − Q H (s, a)|| ∞ ≤ β (3) there exists a protocol that never prunes an optimal action, but prunes all actions so that the agent's mistakes are never more than 4β below optimal. That is, for all times t: V Lt (s t ) ≥ V * (s t ) − 4β, (4) where L t is the agent's policy after t timesteps. Proof of Remark 3. The protocol designer has a β-approximate Q function, denoted Q H , defined as above. Consider the state-specific action pruning function H(s): H(s) = a ∈ A | Q H (s, a) ≥ max a Q H (s, a ) − 2β (5) The protocol prunes all actions not in H(s) according to the self-loop method described above. This protocol induces a pruned Bellman Equation over available actions, H(s), in each state: V H (s) = max a∈H(s) R(s, a) + γ s T (s, a, s )V H (s ) (6) Let a * denote the true optimal action: a * = arg max a Q * (s, a ). To preserve the optimal policy, we need a * ∈ H(s), for each state. Note that a * ∈ H(s) when: Q H (s, a * ) < max a Q H (s, a ) − 2β (7) But by definition of Q H (s, a): |Q H (s, a * ) − max a Q H (s, a)| ≤ 2β (8) Thus, a * ∈ H(s) can never occur. Furthermore, observe that H(s) retains all actions a for which: Q H (s, a) ≥ max a Q H (s, a ) − 2β, (9) holds. Thus, in the worst case, the following two hold: 1. The optimal action estimate is β too low: Q H (s, a * ) = Q * (s, a * ) − β 2. The action with the lowest value, a bad , is β too high: Q H (s, a bad ) = Q * (s, a bad ) + β From Equation  9 , observe that the minimal Q * (s, a bad ) such that a bad ∈ H(s) is: Q * (s, a bad ) + β ≥ Q * (s, a * ) − β − 2β ∴ Q * (s, a bad ) ≥ Q * (s, a * ) − 4β Thus, this pruning protocol never prunes an optimal action, but prunes all actions worse then 4β below a * in value. We conclude that the agent may never execute an action 4β below optimal.
 Experiments This section applies our action pruning protocols (Section 3.3 and Remarks 2 and 3 above) to concrete RL problems. In Experiment 1, action pruning is used to prevent the agent from trying catastrophic actions ("safe exploration"). In Experiment 2, action pruning is used to accelerating learning.
 Protocol for Preventing Catastrophes As noted in the Introduction, human-in-the-loop RL can help prevent disastrous outcomes that result from ignorance of the environment's dynamics or of the reward function. Our goal for this experiment is to prevent the agent from taking catastrophic actions. These are real-world actions so costly that we want the agent to never take the action 2 . This notion of catastrophic action is closely related to ideas in "Safe RL"  [15, 28]  and to work on "significant rare events"  [31] . Section 3.3 describes our protocol program for preventing catastrophes in finite MDPs using action pruning. There are two important elements of this program: 1. When the agent tries a catastrophic action (s, a), a is blocked and the new state and reward is (s, r bad ), where r bad is an extreme negative reward. 2. This (s, a) is stored so that the protocol program can automate the human's intervention, which could allow the human to stop monitoring after all catastrophes have been stored. This protocol prevents catastrophic actions while preserving the optimal policy and having only minimal side-effects on the agent's learning. We can extend this protocol to environments with high-dimensional state spaces. Element (1) above remains the same. But (2) must be modified: preventing future catastrophes requires generalization across catastrophic actions (as there will be infinitely many such actions). We briefly discuss this setting in Appendix A.
 Experiment 1: Preventing Catastrophes in a Pong-like Game Our protocol for preventing catastrophes is intended for use in a real-world environment. Here we provide a preliminary test of our protocol in a simple video game. Our protocol treats the RL agent as a black box. So we applied our protocol to an open-source implementation of the state-of-the-art RL algorithm "Trust Region Policy Optimization" [? ]. The environment was Catcher, a simplified version of Pong with non-visual state representation. Since there are no "catastrophic" actions in Catcher, we modified the game to give a large negative reward when the paddle's speed exceeds a speed limit.  3  We compare the performance of an agent who has assistance by the protocol ("Pruned") and so is blocked from the catastrophic actions to the performance of a normal RL agent ("Not Pruned"). Figure  4  shows the agent's mean performance over the course of learning. We see that the agent with protocol support ("Pruned") performed much better overall. This is unsurprising, as it was blocked from ever doing a catastrophic action. The gap in mean performance is large early on but diminishes as the "Not Pruned" agent learns to avoid high speeds. By the end (i.e. after 400,000 actions), "Not Pruned" is close to "Pruned" in mean performance but its total returns over the whole period are around 20 times worse. Note that the "Pruned" agent observes "odd" state transitions due to being blocked by our protocol. We saw no evidence of these observations having negative side effects on learning.
 Protocol for Accelerating Learning We also conducted a simple experiment in the Taxi domain, introduced by Dietterich  [11] . The Taxi problem is a more complex version of grid world: the agent directs a taxi to each passenger, picks them up, and brings them to their destination. We use Taxi to evaluate the effect of our action pruning protocol for discrete MDPs. There is a natural procedure for pruning suboptimal actions that dramatically reduces the size of the reachable state space: if the taxi is carrying a passenger but is not at the passenger's destination, we prune the dropoff action by returning the agent back to its current state with -0.01 reward. This prevents the agent from exploring a large portion of the state space, thus accelerating learning.
 Experiment 2: Accelerated Learning in Taxi We evaluated Q-learning  [41]  and R-MAX  [5]  with and without action pruning in a simple 10 × 10 instance with one passenger. The taxi starts at (1, 1), the passenger at (4, 3) with destination (2, 2). We ran standard Q-Learning with ε-greedy exploration with ε = 0.2 and with R-MAX using a planning horizon of four. Results are displayed in Figure  5 . Our results suggest that the action pruning protocol simplifies the problem for a Q-Learner and dramatically so for R-Max. In the allotted number of episodes, we see that pruning substantially improves the overall cumulative reward achieved; in the case of R-MAX, the agent is able to effectively solve the problem after a small number of episodes. Further, the results suggests that the agentagnostic method of pruning is effective without having any internal access to the agent's code.
 Conclusion We presented an agent-agnostic method for giving guidance to Reinforcement Learning agents. Protocol programs written in this framework apply to any possible RL agent ("agent-agnosticism") so sophisticated schemes for human-agent interaction can be designed in a modular fashion without the need for adaptation to different RL algorithms. We presented some simple theoretical results that relate our method to existing schemes for interactive RL and illustrated the power of action pruning using one such protocol on two toy domains. In the future, we plan on exploring state manipulation protocols, which offer a powerful avenue for guiding an agent's learning process. For instance, one could imagine a protocol dynamically imposing abstractions that hide confusing features from the agent and encourage efficient learning. Additionally, we want to investigate whether certain types of value initialization protocols can be captured by protocol programs, such as the optimistic initialization for arbitrary domains developed by Machado et al.  [26] . Figure 1 : 1 Figure 1: A general setup for RL with a human in the loop. By instantiating P with different protocol programs, we can implement different mechanisms for human guidance of RL agents.
 Algorithm 4 5 45 Action pruning 1: ∆ = {} Pruned: S × A → A × R 2: r * = ∅ Reward if previous action pruned 3: procedure PRUNEACTIONS(s, r) 4: r = r if r * = ∅ else r * 5: a = L(s, r) 6: if (s, a) ∈ ∆ then 7: (a, r * ) = ∆[(s, a)] r * ) = H(s, r, a) s, a)] = (a, r * ) Reward manipulation 1: procedure MANIPULATEREWARD(s, r) 2:
 Figure 3 : 3 Figure 3: The human allows movement from state 34 to 33 but blocks agent from falling in lava (at 43).
 Figure 4 Figure 5 : 45 Figure 4: Preventing Catastrophic Speeds
			 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
			 The human may need to wait for the start of an episode to do the transfer. Otherwise the agent may experience an impossible state-transition.
			 We allow an RL agent to take sub-optimal actions while learning. Catastrophic actions are not allowed because their cost is orders of magnitude worse than non-catastrophic actions. 3  In this toy environment, a human observer could easily recognize catastrophic actions: they just need to watch the speed of the paddle.


Abstract: Rapid developments are currently taking place in the fields of artificial intelligence (AI) and biotechnology, and applications arising from the convergence of these 2 fields are likely to offer immense opportunities that could greatly benefit human health and biosecurity. The combination of AI and biotechnology could potentially lead to breakthroughs in precision medicine, improved biosurveillance, and discovery of novel medical countermeasures as well as facilitate a more effective public health emergency response. However, as is the case with many preceding transformative technologies, new opportunities often present new risks in parallel. Understanding the current and emerging risks at the intersection of AI and biotechnology is crucial for health security specialists and unlikely to be achieved by examining either field in isolation. Uncertainties multiply as technologies merge, showcasing the need to identify robust assessment frameworks that could adequately analyze the risk landscape emerging at the convergence of these 2 domains. This paper explores the criteria needed to assess risks associated with AI and biotechnology and evaluates 3 previously published risk assessment frameworks. After highlighting their strengths and limitations and applying to relevant AI and biotechnology examples, the authors suggest a hybrid framework with recommendations for future approaches to risk assessment for convergent technologies.

MS John T O'brien
MBBS, MPH Cassidy Nelson

Assessing the Risks Posed by the Convergence of Artificial Intelligence and Biotechnology

Introduction
 Technological Progress S ince the explosion of the information age, the collection of data has been increasing. About 2.5 quintillion  (10 18  ) bytes of data are generated each day, and 90% of all data have been generated in the past few years alone.  1  Although valuable information is often hidden in these vast datasets, due to their size and complexity, a human analyst could take weeks, months, or even years to discover them, if they are able to at all. If one can analyze a large enough set of data, it is possible to infer corollary relationships or patterns that may be of salience. In the past, it was not conceivable for extremely large datasets, or big data, to have been meaningfully analyzed due to the daunting task of doing so manually. Partially as a result of the increasing prevalence of big data requiring new tools for analysis and increased computing power enabling their development, the use of artificial intelligence (AI) in a variety of fields, including the biological sciences, has gained popularity in the past few years. AI can be defined as a system's ability to correctly interpret external data, to learn from such data, and to use those insights to achieve specific goals and tasks through flexible adaptation.  2  Machine learning, a technique that has enabled rapid advancements in the field of AI, is a subset of AI that is heavily based on pattern recognition and describes methods that help computers learn without explicitly being programmed to do so.  2  Deep learning, in turn, is a machine learning technique that arose in the early 2010s and uses computational models that exhibit similar characteristics to the hierarchal information processing in the human brain.  3  This technique is partially responsible for the increased power potential of machine learning algorithms and has been a large contributor to the recent surge of AI research.  4  Alongside AI, developments in biotechnology and the life sciences have progressed rapidly in the past several years. In fact, some biotechnological advancements may even be outpacing Moore's Law for computers.  5  Critical advances in biology such as next-generation DNA sequencing, de novo DNA synthesis, gene editing, genomics, and bioinformatics may result in the life sciences becoming a principal component of the next technological revolution. Given recent progress and projected trajectories, AI and biotechnology have both been described as key pillars of the fourth industrial revolution, with a growing number of technologies and applications arising at their point of convergence every year.  [6] [7] [8] [9]
 Technological Convergence In tandem, AI and biotechnology increase the potential capability of the life sciences and lower the tacit knowledge required to perform previously tedious laboratory tasks. Widening the pool of actors will almost certainly accelerate technological developments, but this could also pose security risks. With increased access to these technologies, so comes the potential for a resulting high-consequence biological event, some of which may have such large consequences that they constitute a global catastrophic biological risk.  10  The risks posed by the convergence of AI with the biological sciences are plentiful and varied, ranging from AI-assisted identification of virulence factors to in silico design of novel pathogens, which could increase the probability of a global catastrophic biological risk event if accidently, carelessly, or deliberately released. For the purposes of this paper, convergence is defined as the technological commingling between the life sciences and AI such that the power of their interaction is greater than the sum of their individual disciplines. That is, the convergence of these 2 fields has a multiplicative effect. This interaction mostly manifests as AI-developments influencing advances, as well as new risks, in the life sciences and biotechnology rather than the other way around. A more comprehensive understanding of such emerging threats is highly important for the field of health security but demands rigorous examination and a systematic approach. In this paper, risk assessment for converging technologies will be discussed and evaluation of 3 existing biosecurity risk assessment frameworks undertaken. After analyzing their strengths and limitations, suggestions will be made regarding how such frameworks could be adjusted to be applicable to the risks arising at the convergence of AI and biotechnology. Some illustrative examples from the converging risk landscape will be discussed, followed by a proposed hybridized framework and key recommendations to begin addressing gaps in current health security risk assessment for converging technologies.
 Risk Assessments For Converging Technologies
 Challenges Assessing Emerging and Converging Risks Risk assessment for emerging technologies is an inherently difficult task and general frameworks are needed for evaluating the risks associated with technological convergence, including AI and biotechnology. Many frameworks are overly specific to a particular scenario or technology (such as narrowly focusing on synthetic biology), contain many assumptions and guesswork, and are not easily applied to other converging or emerging technologies. This is largely because different frameworks serve different purposes, so expectations that they can be implemented more generally are limited. However, the primary strength of a more generic framework for biosecurity risk assessments would be the ability to compare across different technologies or scenarios. Decision makers could then use such frameworks to be more informed as to which risks may be of higher priority. Compared to assessments of singular domains, analyzing risks of converging technologies presents an even greater challenge due to the added layers of complexity and the presence of multidisciplinary components. Converging risks such as those described herein are uniquely challenging to assess because their uncertainties multiply such that one must consider all possible outcomes of AI development in tandem with all possible outcomes of developments in the life sciences. For example, the evaluation criteria in a scenario-based risk assessment for virtual laboratories or 'cloud labs' that are too focused on the biological components may miss important bottlenecks, risks, or gaps that exist within the underlying AI components of the technology. Full assessment may not always be possible for some unpredictable technological advancements. Challenges also arise due to limited access to data and the inability to quantitatively assess emerging risks. Many frameworks include the potential adversary as a component to calculate risk.  11, 12  However, the adversary has a large range of potential attributes that are difficult to capture in their entirety. The adversary could be a nation, a nonstate actor, or an individual. When evaluating the likelihood that one of these adversaries could carry out ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS an attack, assumptions must be made and variables considered on a spectrum. For example, a nation-state could be a low-resourced or well-resourced country. An individual could be anyone from an undergraduate biology student to a senior research professor. Such approaches fail to capture the range of possibilities and thus suffer from decreased granularity. Along these lines, without access to classified information, it is impossible to appreciate a full picture of adversary capability and intent. Furthermore, frameworks are sometimes focused on a technology's ability to aid in a specific element in the development process of a biological agent, which then makes it difficult to assess the risks of a full scenario in which it was implemented successfully. Conversely, assessments that solely focus on a full scenario fail to accurately capture the broad range of potential misapplications of a technology and consequently suffer from excessive specificity. After searching published literature, we identified 3 previously employed frameworks that inform the development of an assessment for converging technologies. These frameworks are examined in this paper in detail, with their strengths and limitations explored.
 Overview of Existing Frameworks The AAAS-FBI-UNICRI Framework The first framework was published in the 2014 report, National and Transnational Security Implications of Big Data in the Life Sciences, and was developed by a joint team consisting of members from the American Association for the Advancement of Science (AAAS) Center for Science, Technology, and Security Policy, the Biological Countermeasures Unit of the Federal Bureau of Investigation (FBI) Weapons of Mass Destruction Directorate, and the United Nations Interregional Crime and Justice Research Institute (UNICRI).  11  The main elements of the AAAS-FBI-UNICRI framework are highlighted in Table  1 . A strong component of this framework is the inclusion of security specialists, in addition to experts from multiple disciplines to evaluate the risks. While being one of the only attempts at risk assessment of a converging technology, the framework is limited in generality because it uses a scenario process to assess risks and its evaluation criteria are heavily skewed to big data and do not neatly map on to other converging technologies such as AI, despite many commonalities between them. The lack of generality is best illustrated by the framework table itself in which the ''Needed scientific expertise and skills'' block is divided into 2 sections to cover all 3 scenarios evaluated. This may be due to the fact that the working group developed the risk scenarios prior to developing the framework itself, resulting in a framework with a narrow scope. When attempting to implement a scenario where a malicious actor uses AI-enabled cloud labs to synthesize a pathogen in silico, the narrowness of this framework becomes more evident. The parameter, ''vulnerabilities in data repositories, software, and/or underlying cyber infrastructure''  11  does not map onto this scenario since none of those 3 vulnerabilities are necessarily present in the case of cloud labs accessed through legitimate channels, albeit for nefarious purposes.
 The NASEM Framework The second framework emerged from the National Academies of Science, Engineering, and Medicine (NASEM) 2018 report, Biodefense in the Age of Synthetic Biology. The NASEM framework (Table  2 ) is strong in that it actively considers the synergy with other technologies as a factor to calculate level of concern. The report also considers machine learning and automation as potential technologies that could facilitate the development of a biological agent. This framework, however, excludes actors who may be beneficial, considering the broad range of potential adversaries. The limitation in using this framework for converging AI and biotechnology risks is that the scope is too specific and restricted to risks from synthetic biology and technologies relevant to the design-build-test cycle. The authors of the report are transparent about these limitations and recognize that every factor may not map on to a specific technology. If one wanted to apply this framework to evaluate the level of concern posed by risks from the convergence of AI and biotechnology, they could face difficulties since the questions used to evaluate the factors are heavily skewed toward synthetic biology. However, since the framework was developed to assess capabilities of a technology to enable harm more   12  This framework (Table  3 ) benefits from including a governability assessment that is used if the average value of the factors from the risk assessment portion are ''medium'' or ''high.'' While assessing governability for approaches to governance has unquestionable value, it can also be an indicator for probability. If a technology is less governable, it is likely to have a higher chance for misuse and resulting risk. A potential weakness in Tucker's framework is the use of an ordinal scale with just 3 values (high, medium, or low). A spectrum spanning a range of only 3 could result in lost information and decreased specificity, which may have negative downstream effects in determining which technologies pose the greatest or lowest risk. Another potential shortcoming is that the framework considers the imminence of the technology to be of equal weight to the other factors in calculating risk. As a result, it could potentially fail to accurately categorize a long-term, high-consequence risk as one worth governing. In a 2017 working paper, Koblentz et al expanded on Tucker's framework, proposing the Tucker-Koblentz framework, which includes the range of applications for misuse as an indicator of risk.  14  This is particularly useful in the context of the converging technologies since they tend to have synergistic effects that increase the range of appli-cations. The authors also noted how risk assessments are implicitly impacted by the anticipated benefits of a technology since such technologies are more likely to be intensely pursued by researchers and private companies, which results in accelerated innovation and diffusion of knowledge. This perspective is useful in the context of the convergence of AI and biotechnology since the range of beneficial applications is rather large.
 The Converging Risk Landscape The overlap between AI and biotechnology is already considerable, with increasing integration of AI into various facets of the biological sciences likely to continue for the foreseeable future.  9  This poses a challenge for the health security risk assessment frameworks previously described and requires further understanding of the risks that need to be prioritized for examination. This section provides an overview of some of the intersections of AI and biotechnology that could be considered for future risk assessment exercises. These sections are presented as illustrative examples and are not meant to propose a taxonomy of risks.
 Synthetic Biology Increasingly, engineering principles are being applied to biology for research and manufacturing purposes. Synthesis and design of existing and novel microorganisms have many legitimate and beneficial applications; however, these same technologies could be used to engineer pathogens for malicious use. AI and machine learning have the potential
 ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS to greatly aid in the design-build-test stages of the synthetic biology process.  15  Current DNA synthesis methods and gene editing technologies, such as the CRISPR-Cas9 system, still pose a high technical barrier for viable pathogen creation. Nonetheless, several recent controversial experiments have showcased the ability to de novo synthesize live viruses, including horsepox and the 1918 flu.  16  Research on emerging techniques, such as enzymatic synthesis of DNA, aims to revolutionize the field of synthetic biology in the next decade, with the end goal being benchtop printers able to synthesize gene-length sequences of DNA to completion.  17  Advances in this space are not only increasing the potential power and capability of synthetic biology but are also lowering the technical barriers and tacit knowledge required for its widespread use.  18  Within the realm of synthetic biology, AI could potentially lower some of the barriers for a malicious actor to design dangerous pathogens with custom features. It is worth noting, however, that AI will have a differential effect on technological hurdles, with some bottlenecks remaining difficult to traverse. For a more detailed examination of how the difficulty of design steps can affect the probability of a successful biological attack, we recommend the paper by Sandberg and Nelson also published in this issue.  19  AI has enabled a substantial reduction in the time required to design and test functionality in fields such as protein design.  20  It may also lower barriers in the design stages of the design-build-test cycle and could be used to identify optimal mutations of a pathogen that enhance virulence factors, expand host range, alter routes of transmission, encode resistance to medical countermeasures, and intensify mortality.  21  Genetic changes could be identified and made to a virus that is traditionally benign in nature, allowing it to bypass screening procedures currently used by mail-order DNA synthesis services. Similarly, AI could be leveraged to circumvent field detection technologies by identifying code sequences that could be altered without changing function.  11  Deep learning could potentially identify genetic functions that code for vulnerabilities and interconnections between the immune system and microbiome. It may be possible that malicious actors could leverage this information to engineer ''precision maladies,'' or pathogens that target important mechanisms the immune system or the microbiome of specific subpopulations.  22  It is important to note, however, that simply identifying certain vulnerabilities does not mean actors can then freely make changes, significant bottlenecks would still have to be traversed in the overall development process and the feasibility of targeting specific subpopulations would remain an unknown. Moreover, in the past few years, several studies have been published on predicting the pathogenic potential of natural and synthetic DNA using AI. Although improving biosecurity is one of the motives for this research, the results of this work may have the reverse effect and pose information hazards if the methods and algorithms are published. PaPrBaG and DeePaC are examples of machine learning algorithms predicting pathogenic potential that have made all source code available online.  23, 24  These algorithms could potentially aid a bad actor with the intent of designing a pathogen that can cause maximal harm. AI can not only assist in the design step of the process, but it can also, to a lesser degree, support the building and testing phases. The rise of AI-enabled cloud labs to automate procedures that used to require hands-on, tacit knowledge to perform tasks could further widen the pool of actors who could perform certain biology experiments. Cloud labs could allow a malicious actor to perform synthesis and carry out subsequent testing undetected and with greater ease.  25  The next evolution of cloud labs may further enable AI scientists capable of carrying out experiments in entirety by developing and testing hypotheses, interpreting results, and amending hypotheses for retesting, thus lowering the technical barrier even further.  26  Furthermore, directed evolution of pathogens, which spans all 3 phases of the design-build-test cycle, is another route a malicious actor could take in the development of a customized pathogen.  13  AI techniques have already been leveraged to accelerate this process and guide directed evolution in protein engineering.  27  Overall, in the future, it may be possible for a bad actor to perform complete hands-off, in silico design, building, and testing of a novel or recreated pathogen by leveraging the technologies and advancements outlined above.  25, 28  The potential democratization and synergy of these technologies pose potentially high-consequence risks that could elevate to a globally catastrophic level if not recognized and mitigated.
 Vulnerabilities and AI Systems As technologies in the biological sciences continue to be developed, AI-enabled offensive cyber-attacks may become an increasing biosecurity threat. For example, cybersecurity vulnerabilities in machines such as DNA synthesizers, could allow the introduction of malware to corrupt the design or audio-record printed DNA sequences, disrupt laboratory biosecurity, or allow intruder access to sensitive information.  29, 30  In addition, AI system vulnerabilities could also have downstream effects on human health. If healthcare becomes increasingly automated with the use of AI, so does increased risk for cyber-intrusion and subsequent manipulation of medical devices, which could result in harm for the end-user. TrapX Security Labs notes in their latest MED-JACK report  31  that many medical devices and equipment in use today, such as infusion pumps, medical lasers, LASIK surgical machines, and life support equipment are vulnerable to hijacking by malicious actors. Wireless connectivity is becoming increasingly common in implantable medical devices, opening up the risk of malicious actors mounting attacks from anywhere in the world.  32
 O'BRIEN AND NELSON Beyond the risks associated with medical device exploitation, it is possible that in the future computer systems will be integrated with human physiology and, therefore, pose novel vulnerabilities. Brain-computer interfaces (BCIs), traditionally used in medicine for motor-neurological disorders, are AI systems that allow for direct communication between the brain and an external computer.  33  BCIs allow for a bidirectional flow of information, meaning the brain can receive signals from an external source and vice versa. The neurotechnology company, Neuralink, has recently claimed that a monkey was able to control a computer using one of their implants.  34  This concept may seem farfetched, but in 2004 a paralyzed man with an implanted BCI was able to play computer games and check email using only his mind.  35  Other studies have shown a ''brainbrain'' interface between mammals is possible.  36, 37  In 2013, one researcher at the University of Washington was able to send a brain signal captured by electroencephalography over the internet to control the hand movements of another by way of transcranial magnetic stimulation.  38  Advances are occurring at a rapid pace and many previous technical bottlenecks that have prevented BCIs from widespread implementation are beginning to be overcome.  [39] [40] [41]  Research and development of BCIs have accelerated quickly in the past decade.  42  Future directions seek to achieve a symbiosis of AI and the human brain for cognitive enhancement and rapid transfer of information between individuals or computer systems.  34  Rather than having to spend time looking up a subject, performing a calculation, or even speaking to another individual, the transfer of information could be nearly instantaneous. There have already been numerous studies conducted researching the use of BCIs for cognitive enhancement in domains such as learning and memory, perception, attention, and risk aversion (one being able to incite riskier behavior).  43  Additionally, studies have explored the military applications of BCIs, and the field receives a bulk of its funding from US Department of Defense sources such as the Defense Advanced Research Projects Agency.  44  While the commercial implementation of BCIs may not occur until well into the future, it is still valuable to consider the risks that could arise in order to highlight the need for security-by-design thinking and avoid path dependency, which could result in vulnerabilities-like those seen with current medical devices-persisting in future implementations. Cyber vulnerabilities in current BCIs have already been identified, including those that could cause physical harm to the user and influence behavior.  [45] [46] [47] [48]  In a future where BCIs are commonplace alongside advanced understandings of neuroscience, it may be possible for a bad actor to achieve limited influence over the behavior of a population or cause potential harm to users. This issue highlights the need to have robust risk assessment prior to widespread technological adoption, allowing for regulation, governance, and security measures to take identified concerns into account.
 Combined Illustrative Framework Given the rapid developments in AI and biotechnology and the convergence of these 2 areas, existing risk assessment frameworks could be adapted to properly examine emerging threats. Table  3  is an illustrative framework that adapts the original AAAS-FBI-UNICRI framework by removing some elements while adding some from other frameworks to improve its capability to assess converging risks, such as those posed by AI and biotechnology. Convergent risk scenarios from this paper are included as examples of how this framework might be implemented; the completed boxes presented in the table are illustrative only and not meant to be conclusive, as there was no large-scale expert review and consensus. In Table  4 , the adversary spans the same range for each element, which may suggest it is a variable that might not be worth including or at least reconsidered. However, a comprehensive assessment by multidisciplinary experts with multiple scenarios would need to be conducted to determine this. In this framework, rather than using an ordinal scale of only 3 (low, moderate, high) to assess each parameter, an ordinal scale of 5 (very low, low, moderate, high, very high) was used to increase specificity and better delineate between scenarios. Each parameter is meant to be evaluated based on the projected timeline. For instance, in the case of BCIs, it is assumed the parameter will be highly accessible and democratized in the long-term. Governability was included as a component contributing to probability since a technology convergence that is less governable may be more likely to be maliciously used. Democratization, called ''accessibility'' in other frameworks, was included because a technology that is more accessible to a wide range of individuals is assumed to have higher potential for misuse. The remaining evaluation criteria (vulnerabilities, needed skills and expertise, magnitude of potential consequences, and existing countermeasures) were selected because they are common to many of the frameworks reviewed herein and we believe are integral to any risk assessment framework. ''Vulnerabilities'' is meant to describe and illuminate any gaps that could be exploited to cause harm. ''Needed skills and expertise'' is meant to describe the scope of potential adversaries and help identify potential bottlenecks. ''Magnitude of potential consequences'' and ''existing countermeasures'' are meant to contextualize the scope of potential harm and balance that with any mitigating factors in place. As one might expect, this combined framework has many limitations. For example, depending on the type of pathogen designed or synthesized, the high variance changes the amount of expertise needed and potential consequences over time, as does the nature and sophistication of the type of BCI exploitation. The framework also includes uncertainties related to the timeline and assessing governability, which highlights the high degree of assumptions required to complete and use it. Moreover, the framework lacks a standardized mechanism for evaluating factors in a way that calculates risk. The hope is that explicitly identifying ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS these limitations can help illustrate the gaps and challenges in developing a comprehensive framework for converging technologies that may pose high-consequence risks.
 Recommendations
 Risk Assessment Framework Approaches A risk assessment framework is needed that is applicable to converging technologies in the life sciences that balances generality to capture a broad scope but also maintains enough specificity to capture nuances. It is paramount that such frameworks be developed and implemented by a multidisciplinary team of experts that includes both security officials and scientists. Such a strategy allows for a more comprehensive and, subsequently, more accurate depiction of the potential risks. To further increase robustness, existing frameworks should be iteratively built upon and improved, such as the combined framework presented in this paper. A general framework intended to apply to a diverse range of converging or emerging technologies should be developed prior to determining the scenario to be evaluated. It can then be revised as scenarios or technologies are assessed. This would benefit the applicability of such a framework to other scenarios or technologies rather than narrowly limiting it to a specific issue. The role that time plays in calculating risk for an emerging or converging technology should be weighed. While a technology that is assumed to not be relevant until the far future may not pose an immediate risk, it should still be considered, as it could pose a significant threat once the technology materializes. Additionally, projected timelines for emerging technologies are inherently uncertain and technologies may develop more rapidly than expected. Therefore, it is worth considering projected timelines as a framing element rather than a factor to calculate risk.
 Revisiting the AAAS-FBI-UNICRI Framework As 5 years have passed since publication of the initial AAAS-FBI-UNICRI report, we recommend this working group reconvene to reassess the risks outlined in this paper and update their framework to be more inclusive of other converging threats. As noted in their last report, ''if scientists and security experts can assess the risks and benefits together and on a routine basis, the potential risk that biological [big data] would be used by adversaries decreases.'' We believe that this team is best equipped to develop an iterative set of frameworks related to this topic because the workgroup includes knowledgeable experts from a wide range of disciplines and the issues related to big data and AI are similar in nature. With perspectives from experts in law enforcement, the life sciences, and AI, the team is uniquely positioned to take on such a task. While their initial project focused narrowly on big data, it would be of great value to extend that scope to AI in the life sciences and provide relevant stakeholders in academia, government, and the private sector with an informational risk assessment that will help them make informed research, policy, and security decisions on these topics. This working group can leverage frameworks-such as the NASEM and Tucker-Koblentz frameworks-that have been developed since their initial report to improve upon their own. The Koblentz et al  14  paper could provide them with an insightful review of the strengths and weaknesses of many previous frameworks and would be a useful resource as the working group develops the next iteration. Components from the NASEM framework could be leveraged to increase the generality and broadness of their framework. The illustrative framework presented in this paper (Table  4 ) may serve as a useful starting point for the joint team.
 Conclusion AI and biotechnology are both rapidly growing fields. The biosecurity risks arising at their points of convergence are vast, highlighting the need for comprehensive mapping and thorough, robust risk assessment. Current frameworks used for risk assessment in health security settings may fail to capture the nuanced areas of these converging fields and could benefit from considering generality and timeline for technology adoption. We recommend that a multidisciplinary working group comprised of AAAS, the FBI, UNICRI, and other specialists gather to formulate a framework to comprehensively assess risks the results from converging technologies such as AI and biotechnology as well as others that may emerge in the future. The combined framework presented in this paper could serve as a starting point for the development of more robust biosecurity risk assessments for converging technologies like AI and biotechnology. While this new framework does not solve all of the identified challenges, it explicitly elucidates the gaps in our ability to accurately assess risks associated with emerging and converging technologies and will hopefully facilitate adaption of previous frameworks in order to develop a more inclusive health security risk assessment that properly balances generality and specificity and is regularly and iteratively revisited. Table 1 . 1 Main AAAS-FBI-UNICRI Framework Elements to Evaluate the Risks of Big Data 11   Risk Assessment Probability Consequences Adversary Vulnerabilities in Needed scientific expertise and skills Severe consequences Sufficient existing data repositories, software, and/or underlying cyber infrastructure To exploit vulnerabilities in the system To use big data analytics to design harmful biological agents to economics, political system, society, health, environment, and/or agriculture countermeasures
 Table 2 . 2 Main NASEM Framework Elements to Evaluate Level of Concern of Technologies in Synthetic Biology 14   Level of Concern About the Capability Usability of the Technology Usability as a Weapon Requirements of Actors Potential for Mitigation Ease of use Production and delivery Access to expertise Deterrence and prevention capabilities Rate of development Scope of casualty Access to resources Capability to recognize an attack Barriers to use Predictability of results Organizational footprint Attribution capabilities Synergy with other requirements Consequence management capabilities technologies
 Table 3 . 3 Main Tucker Decision Framework Elements to Assess Misuse and Governability 13   Risk of Misuse Governability Accessibility Embodiment Ease of misuse Maturity Magnitude of potential harm Convergence Imminence of potential misuse Rate of advance International diffusion Note: The elements were determined by averaging the values (high, medium, or low) yielded by the parameters. Governability was determined if risk of misuse is medium or high.
 Table 4 . 4 Illustrative Framework Adapted from AAAS-FBI-UNICI, NASEM, and Tucker Frameworks [12] [13] [14] [15]   Convergent Risk Scenarios In Silico Design In Silico Synthesis Brain-Computer of a Pathogen of a Pathogen Interface Exploitation
			 Volume 18, Number 3, 2020