Abstract: How should scientific funders evaluate research with public health risks? Some risky work is valuable, but accepting too much risk may be ethically neglectful. Recent controversy over H5N1 influenza experiments has highlighted the difficulty of this problem. Advocates of the research claim the work is needed to understand pandemics, while opponents claim that accidents or misuse could release the very pandemic the work is meant to prevent. In an attempt to resolve the debate, the US government sponsored an independent evaluation that successfully produced a quantitative estimate of the risks involved, but only a qualitative estimate of the benefits. Given the difficulties of this ''apples-to-oranges'' risk-benefit analysis, what is the best way forward? Here we outline a general approach for balancing risks and benefits of research with public risks. Instead of directly comparing risks and benefits, our approach requires only an estimate of risk, which is then translated into a financial price. This estimate can be obtained either through a centrally commissioned risk assessment or by mandating liability insurance, which allows private markets to estimate the financial burden of risky research. The resulting price can then be included in the cost of the research, enabling funders to evaluate grants as usual-comparing the scientific merits of a project against its full cost to society. This approach has the advantage of aligning incentives by assigning costs to those responsible for risks. It also keeps scientific funding decisions in the hands of scientists, while involving the public on questions of values and risk experts on risk evaluation.

Sebastian Farquhar
Owen Cotton-Barratt
Andrew Snyder-Beattie

Pricing Externalities to Balance Public Risks and Benefits of Research

I n 2012, multiple research groups conducted experiments that produced strains of H5N1 avian influenza that were airborne transmissible between mammals.  1, 2  Given that this disease is often fatal in humans, these particular ''gain-of-function'' experiments sparked wide controversy.* Advocates of the research argued that the experiments were necessary to improve our understanding of the virus, thus enabling better disease surveillance and vaccine production. Opponents of the research argued that a laboratory accident could trigger a global pandemic, or that the research could be used by malicious actors.  3  The debate culminated in a moratorium on US public funding of research ''that may be reasonably anticipated to confer attributes to influenza, MERS, or SARS viruses such that the virus would have enhanced pathogenicity and/or transmissibility in mammals via the respiratory route.''  4  Meanwhile, the US government also commissioned an Sebastian Farquhar, MPhysPhil, is Project Manager; Owen Cotton-Barratt, DPhil, is a Research Fellow; and Andrew Snyder-Beattie, MS, is Director of Research; all are at the University of Oxford, Future of Humanity Institute, Oxford, England. This article reflects only the view of the authors. The ERCEA is not responsible for any use that may be made of the information it contains. ª Sebastian Farquhar et al., 2017; Published by Mary Ann Liebert, Inc. This Open Access article is distributed under the terms of the Creative Commons License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. *Gain-of-function experiments encompass a wide range of research, and only a small subset of such experiments were subject to this controversy. In this article, we focus on gain-of-function research of concern, by which we mean research of the sort affected by the funding moratorium, as a case study of the more general problem of making research funding decisions for dualuse research of concern. Health Security Volume 15, Number 4, 2017 Mary Ann Liebert, Inc. DOI: 10.1089/hs.2016.0118 independent study from Gryphon Scientific to weigh the benefits of the research against the possible biosafety and biosecurity risks.  5  While their research succeeded in producing a quantitative estimate of biosafety risk, it explained the benefits of the research only in qualitative terms. This controversy highlights the difficulty of evaluating research that has the potential to entail significant public risk. Research that is particularly likely to lead to both positive and negative impacts is especially hard to evaluate.  6  These negative consequences might be either the risk of an accident linked directly to the research itself, or risks of deliberate misuse of the resulting technologies. Avoiding research that has any theoretical risk at all would mean missing out on crucial opportunities. But research communities that underestimate the potential for a significant accident, or assume that people will use information gained through research only for good, may be neglecting their duty to society.  7  The H5N1 flu experiments mark a particularly interesting case study that we use to highlight some problems in evaluating the risks and benefits of dual-use research. { In this article, we discuss some of the current difficulties of existing approaches to evaluating risky research, and we propose a general framework that avoids many of the current problems. Within this framework, we outline 2 possible approaches to implementing the framework: a marketled liability insurance approach, and a state-led approach. We conclude by comparing the approaches with each other and with current policy.
 Difficulties of Risk-Benefit Analysis for Scientific Research Risk-benefit analysis, the approach used in the Gryphon Scientific report, is an analytical tool that has been successfully applied to a wide range of domains.  8  Analysts outline a range of scenarios with positive and negative impacts, estimate the probabilities of those scenarios and the net size of the positive impact, and compute the expected value of the proposal. When attempting to do a riskbenefit analysis of research, determining the scenarios and the probabilities of each scenario can be difficult. But as it turns out, estimating biosafety risks is far easier than esti-mating the benefits of research. The reason for this is that one can outline the potential scenarios for biosafety risks fairly well. In order for a laboratory accident to cause a global pandemic, a set of events must occur. The probability of each of these steps can then be estimated using historical data on laboratory accidents and models of disease spread. These estimates are highly uncertain, because of limitations in available data. The data describing exactly what activities labs engage in and their historical failure rates are incomplete, though some records are kept. Moreover, assumptions are required to extrapolate past records into the future. For example, lab safety standards and equipment have improved over the past several decades, but at the same time new labs are being built under the jurisdiction of regulators with less of a track record establishing their biosafety capabilities. Risk estimators must also make some estimate of the probability distribution of key parameters that cannot be empirically tested. For example, what is the probability that a reassortant virus will be sufficiently transmissible to spread to a wide population? Nevertheless, quantitative estimates are possible, though they lead to disagreement.  [9] [10] [11]  Many risks that are in principle very difficult to estimate for similar reasons are nevertheless quantitatively priced by insurers today-such as terrorism,  12  cyberattack,  13  and harm to patients in clinical trials.  14  Conversely, estimating the benefits of research is extremely difficult. Unlike estimating the probability of a laboratory accident scenario, evaluating the benefits of research involves uncertainty over what the possible scenarios even are. Typically, researchers have little information about the potential results of research-which is one of the motivations to do research in the first place. Even the most experienced researchers find it difficult to estimate the exante value of new research projects. The work recreating the genome of the influenza virus responsible for the 1918 pandemic  15  received some criticism for vaguely promising unspecified future benefits  16  but, after some time, has plausibly created tangible benefits including, perhaps, aiding in the management of the 2009 H1N1 pandemic, according to the authors of the original paper.  17  In their review, Gryphon Scientific did not even attempt to assess the benefits of research quantitatively, and it is this part of the risk-benefit calculation that typically poses the greatest difficulty.  5, 18  Evaluating the risks from malicious activity is more difficult than estimating the risk of accidents, but easier than estimating the benefits of research. Ultimately, the Gryphon Scientific report included a quantitative assessment of some biosecurity risks (though not the risks posed by the information discovered by the research). Unlike the accident scenarios, uncertainties around both the intentions and capabilities of malicious actors mean a wider range of scenarios is possible. But despite the difficulties in quantifying such risk, many insurance companies still offer terrorism insurance, and models do exist for putting rough numbers on such risks.  12
 { In the context of atomic technologies, discussions of dual-use technology focused on the ability to rapidly convert civilian technologies for military purposes,  8  and this trend continued for much of the 20th century.  9  Recently, discussions of dual-use research have broadened to include nonmilitary actors and the risk of accidents related to research.  5, 10  It is this last, broadest sense of dual-use research that we focus on in this article, as it captures the full spectrum of risks resulting from research activities. We acknowledge but do not address the fact that the definition of ''dual-use'' remains unsettled,  11  and that one might want to consider dual-use research, technology built on that research, and artefacts implementing that technology separately.  9
 BALANCING RISKS AND BENEFITS OF RESEARCH When science funders evaluate dual-use research, they are faced with risks and benefits that are uncertain in different ways. Most of the risks are quantitatively estimable, but the parameters are hard to pin down. It is our view that, even when highly uncertain, explicit quantitative risk estimates are preferable to implicit ones, because they allow transparent discussion about disagreements and mitigate human biases in judgments around small probabilities.  19  The benefits, however, are so uncertain that even specifying the scenarios is impossible. Fortunately, existing grantmaking approaches are already designed to compare projects with nebulous benefits. So, as long as the quantified risks are accounted for in the cost of a project, these existing systems are well suited for evaluating this research.
 Existing Approaches and the Scientific Grant-Making Process Grant makers already assess the uncertain benefits of research against their costs and the opportunity costs of unfunded research. Although it is hard to judge quantitatively, reviewers assess the potential for scientific excellence in different proposals. However, the costs that reviewers consider are primarily direct financial costs, rather than externalities such as the risks borne by the public from dual-use research. In principle, research funders prefer to fund portfolios of research with maximal benefits of research for society (including positive externalities) and minimal costs (including negative externalities). The concept of scientific merit already includes some sense of positive externalities, and the UK, at least, also considers the social impact of research and its effect on the research ecosystem.  20  Conceptual tools and intuitions on risk externalities are less well developed. Reviewers often implicitly evaluate dual-use risk as a matter of personal or professional ethics  21  or for the sake of reputation of the field. However, for most research, the dual-use stakes are low. In these situations, researchers have no need to also become experts in risk assessment and management. Research institutions rely instead on safety regulations for particular categories of risk in order to make sure research processes are safe.  22  These tools work very well when most of the risk is local, for accidental releases at the community level, or for health risks to lab researchers themselves. Historically, this is where most of the risk in biological research has lain and is therefore an entirely appropriate default. However, in the case of risks that carry a small chance of large catastrophe (eg, research that creates potential pandemic pathogens), these regulations are not enough. Some funders, such as the US government, provide additional decisionprocess guidelines for funding research involving specific pathogens, like H5N1,  23  as well as institutional guidelines for best practice for dual-use research.  24  Others, such as the UK's Biotechnology and Biological Sciences Research Council, the Medical Research Council, and the Wellcome Trust, require grant applicants to consider near-and medium-term risks of misuse but rely heavily on self-governance and a responsible research culture.  25  These frameworks put a high burden on those pursuing dual-use research to become risk assessors. Moreover, public funding guidelines only affect government funding and cooperative private funders. Private dual-use research is less constrained.  26  Some additional judgment can in principle be applied at the stage of publication,  27  but a 2012 survey of 127 life sciences journals found that none reported refusing a submission on biosecurity grounds.  28  Export controls may also be able to restrict some kinds of publication  29  in some jurisdictions. It may be that under the Australia Group's ''no undercut'' principle, in its guidelines for transfers of sensitive chemical or biological weapons, if any participant nation were to block publication using export controls then all would be obliged to (though the mechanisms for applying the guidelines to research results is unclear).  30
 Pricing and Using Externalities in Grant Making
 Absolute Risk as a Cost of a Grant Hard-and-fast rules about what research techniques are too risky, without considering the benefits, may not be responsive enough, because more important research merits more risk.  31  Our principal proposal is that one could assess and price absolute expected risk and explicitly include this cost in grant proposals. This helps resolve the fact that while risks can be comparatively easy to assess, the benefits of research are extremely challenging to quantify. Our proposal allows grant reviewers to use existing frameworks to evaluate grant proposals but weigh potential benefits against a fairer reflection of their social costs. It also follows calls to strengthen the evaluation of risks at pre-funding stages of the research process.  32  For now, we set aside the issues of where an independent estimate of the cost of externalities comes from and where the money to cover this explicit cost is paid. The 2 main approaches to setting estimates, which we consider in detail in the next section, are: 1. To establish clear liability in case of catastrophe and require grant holders to purchase liability insurance as part of the grant. The strength of this is that it is a market-led approach, with insurers incentivized to price externalities correctly. 2. To centrally commission absolute risk assessments to price the externalities, and to require a payment to a state or non-state body to cover the expected cost. The strength of this approach is that it works even if there may be no clear liability after the fact, so it could address biosecurity as well as biosafety risks.
 Benefits of the Approaches Both approaches have 4 main advantages. First, they give decision makers greater incentive to use an accurately priced risk when deciding what research to fund. Currently, the costs of accidental or deliberate harm from gain-offunction research of concern are borne by members of the public in any country as well as lab workers. For example, after a foot-and-mouth disease outbreak in Pirbright in 2007, caused by poorly maintained drainage,  33  prosecutors were unable to seek damages because there was no demonstrable negligence of a specific party.  34  Instead, governments bore the financial cost of compulsory slaughter, farmers bore the costs of damaged and restricted business, and consumers experienced higher food prices. Where the risks of an activity are not borne by those making decisions about the activity, economic theory predicts that more risk will be taken than is socially optimal.  35  So long as the risks are priced correctly and the transaction costs are sufficiently low, research institutions that pay upfront for the risks of research are more likely to engage in socially optimal risktaking. If implemented successfully, this would result in both better grant choices and better use of risk reducing techniques and equipment. Second, these approaches keep decisions about what research gets funded in the hands of researchers. One of the largest difficulties in assessing the social value of research is assessing the intellectual benefits of research. Researchers themselves are best placed to evaluate the potential for success and the intellectual merit of research because they have the deepest knowledge of the area. Third, these approaches incorporate democratic or public input on the significance of risks. Although researchers have expertise in the potential for intellectual success from a grant proposal, they are not necessarily able to speak for the values of society. Researchers have an obligation to incorporate public input on value systems and consider the justice implications of their work.  31  Our proposal creates ways for the public to express their values without creating rules allowing or disallowing certain research techniques or areas of inquiry. In the case of an insurance mechanism, the legislative branch effectively determines the social cost borne were the risk to materialize, via the judiciary. In the case of a central body setting the costs, this body would ultimately be accountable to the electorate. Fourth, these approaches ensure appropriate risk assessment and governance expertise is brought to bear. Researchers submitting grant proposals are often asked to consider potential dual-use and accident risk. Grant makers do so similarly. However, neither group tends to include specialists in risk assessment or biosecurity.  26  If absolute risk is priced separately, the price can be set by expert risk assessors. Throughout, our guiding principle is to ensure the right incentives are in place and that components of the funding decision are broken up such that those best placed to assess them do so.
 Disadvantages of the Approaches Although we believe these approaches might improve the capacity of the research funding system to manage catastrophic risk, they introduce some new concerns. First, research institutions currently receive a subsidy in the form of an implied guarantee covering the costs of imposed risks. If they were asked to pay for this guarantee, it would represent an additional unfunded expense. Since the aim of this proposal is to distribute research effort within, say, the life sciences rather than to change the absolute amount of life science work being done, one might increase funding for life sciences research to compensate. This would appear to increase explicit expenditure in government research budgets, but in fact moves an implicit, random, and large subsidy from the budgets of disaster response agencies, who do not make decisions that determine the risk exposure, to an explicit, predictable, and moderate payment by those who do make such decisions. For example, governments and consumers bear a large part of the cost of escapes of animal pathogens from research labs  36  but have less influence over risk exposure than research funding bodies. Second, penalizing laboratories for reporting accidents and near misses in a timely way harms biosafety and biosecurity in the long run. Increasing reporting would make it easier to use lessons from mistakes to improve lab design and improve accident response. Mechanisms for pricing risk will work best if they avoid creating perverse incentives around reporting. Options for facilitating this include simply not using the information from recent accidents in the case of central analysis, or forbidding a premium hike in the case of liability insurance requirements. Third, these approaches increase bureaucratic overhead. There are several things that might be done to reduce this overhead. One could apply our proposal only to areas with particularly high risk of low-probability but high-impact events. For example, it might be applied only to the gainof-function research that has been covered by the recent US moratorium. Or rather than carrying out a separate risk analysis for each proposal, one could establish broad categories of similar work and set costs for that category as a whole. For some categories, it might be possible to get involvement from intelligence communities to incorporate information not normally available. In the future, scope could be expanded by addressing new areas or by making the cost estimates more granular.
 Making Decisions Under Uncertainty There are things these approaches would not fix. Risk in many cases of dual-use research is very uncertain. No risk management or risk governance process can solve this entirely. It is hoped that in the case of market-led approaches, the profit motive of companies providing insurance as well as an awareness of ''winners' curse'' dynamics will incentivize BALANCING RISKS AND BENEFITS OF RESEARCH appropriate pricing. In the case of a state-led approach, it is hoped that the system will increase the capacity for riskassessment by developing a core of expert assessors. Political pressures or ''short-termism'' might cause prices to be inaccurate. However, this is also a feature of existing approaches to dual-use risk management.
 Approaches to Risk Pricing
 Mandatory Liability Insurance Laboratories conducting experiments of appropriate kinds could be mandated to purchase insurance against liability claims arising from damaging accidents from their research. Insurance companies would then pay out to injured parties in the public in case of an accident. Some proposals in this direction have already been discussed in the UK.  27  In some cases, appropriate use of liability for damages is more effective than regulation in encouraging risk-reducing behavior.  37  In most jurisdictions, negligence leading to catastrophic damage would already establish liability. Sometimes, the complexity of the research process makes it hard to establish negligence of a specific party.  34  Governments might therefore need to legislate to establish clear duty of liability in any damages caused by accident, even if there was no negligence. Liability would give universities strong incentives to minimize the risk. Framers of this legislation would need to be quite specific about which sorts of liability were included (eg, health damage) and which were not (eg, adversely affected tourism, perhaps, or terrorism). Such an approach might be similar to some aspects of the 1957 Price-Anderson Act, which placed a strict liability on operators of potentially risky private nuclear facilities and a mandate to purchase the highest commercially available amount of insurance coverage.  38  A similar approach has been tried and advocated for in environmental risk, though it proved hard to maintain a healthy insurance market.  39, 40  Requiring insurance is important because some risk takers will not be able to compensate victims for catastrophic harms. Many universities currently self-insure against the damage of accidents in their research. Where there is a small chance of catastrophic damage, they do not have enough assets to cover the damages-they are ''judgment-proof.'' Insurance providers, with much greater financial liquidity, are able to bear the full costs even when payouts are very large, which incentivizes them to set premiums appropriately.  41  For the most catastrophic forms of damage, such as a global pandemic, costs would be too big even for a reinsurer to bear. As a result, an insurance approach could involve an explicitly negotiated liability cap, in which case regulatory bodies tasked with managing the most extreme forms of risk are essential. Various amendments to the Price-Anderson Act demonstrate the problems facing liability limits. Indemnifying those creating the risk from all costs above the privately insurable can cause excess risk taking.  42  The most recent version of Price-Anderson requires regulated nuclear energy providers to pool to jointly cover more than $10 billion of costs beyond that covered for each individual party by insurers. While this protects the public by identifying a source of funds beforehand, the fact that the individual contributions do not need to be set aside in advance means that the entire industry carries a large, uninsured, systemic risk. A modern system based on catastrophe bonds might offer a safer way to increase the feasible levels of financing.
 Advantages of the Liability Approach There are 2 main advantages to this market-led approach. First, it is a relatively light intervention, requiring little ongoing work from the state (other than maintaining a healthy insurance market). Second, the insurers would have a profit incentive to accurately estimate the risk, reducing possible politicization of the risk assessment process. In a sufficiently liquid market, competitive pressures should bring premiums down. But insurers are aware of a ''winners' curse'' in the insurance market and would avoid setting premiums too low to cover the long-term risk. If insurers adjust their premiums when safer systems and equipment are adopted, scientists and engineers would be incentivized to devise effective safety protocols to reduce their institutions' insurance costs. This would create a financial driver for applied biosafety and biosecurity research. Imposing liability has improved outcomes in other domains such as occupational safety, medicine, and general risk management in nonprofits and government agencies.  43
 Possible Issues with the Liability Approach Insurers must be willing to insure against biosafety and biosecurity risk. While insurers have shown some interest,  44  there is no functioning market for the same 2 main reasons insurers were unwilling to take on nuclear risk before the 1957 Price-Anderson Act:  38  First, the potential risks are simply too large. A catastrophic global pandemic could kill hundreds of millions of people, and even the largest reinsurers would be unable to absorb this cost without bankrupting themselves (costs above this level will be implicitly backed by the state or the public in any case). It is better to be explicit and cap liability at a specific industry-wide figure. If the cap were sufficiently large, the effect should be more appropriate risk aversion, even if the tail risk for the insurer were not fully internalized. Second, the risks are hard to model and the market would be quite small. Developing models to estimate the risk could be more costly than the expected profit from being in the market. However, insurers do already have models for hard-to-anticipate risks such as terrorism and global pandemics. If need be, the development of appropriate models to facilitate this insurance could be explicitly subsidized. Whether or not insurers are willing to take on the risk, there are challenges to international adoption. Liability laws differ across jurisdictions, which would particularly affect international collaborations. One effective work-around might be to form an agreement among top-tier research journals to require proof of suitable liability insurance for some types of papers. { Liability insurance can also increase moral hazard, by making actors less responsible for the consequences of their actions. This effect could be reduced if the excess on the insurance were large enough that research institutions still had a direct incentive to avoid risks. Finally, using a liability approach to capture biosecurity risks would be difficult. If a terrorist group used information they gleaned from a paper published by a research group, establishing liability would require lengthy legal dispute. Moreover, done carelessly, the uncertainty additional but unclear liability could create might have an inappropriate chilling effect on research.
 Centrally Commissioned Risk Assessments The second approach is to centralize risk assessments. When an area of potential concern is identified, a body commissioned by the state would perform an analysis of the risks involved. This might be similar to the recent Gryphon Scientific analysis, except that it would not attempt to analyze the benefits. This absolute risk analysis would present its outcomes in monetary terms, using value of statistical life figures to convert into a cost. In order to do work of a priced type, laboratories would have to pay the appropriate charge to a central authority. This charge could cover part of the risk of biosafety and biosecurity preparation as well as the cost of risk assessment.
 Advantages of the Centrally Commissioned Risk Assessment Compared to the market-led approach, centralizing the risk assessments has 2 main benefits. First, it can be done by fiat without needing to persuade insurers to enter and stay in the market. Second, it works well even in cases where there will be no clear causal chain establishing liability, such as for biosecurity risks (including risks from the misuse of information produced from such research). Funds are collected based on the central assessment of the size of the risk posed by the research, with no need to work out after the fact which specific group is liable. The Gryphon Scientific report concluded that the biosecurity risks looked at least as large as the biosafety risks, so this is a significant benefit in the case of gain-of-function research of concern.  5  Third, at least in principle, avoiding the need for courts to establish liability might speed up payments.
 Possible Issues with the Central Approach Patchy implementation only in certain jurisdictions might just move research from nations with good risk management processes to those without. In practice, however, many researchers are likely to want to remain at top labs, limiting the risk of research simply moving to jurisdictions that do not assess the externalities of research. There may be a need for special arrangements for research done in partnership with labs outside the collecting jurisdiction, such as applying a requirement for proof of insurance at the stage of publishing. The risk assessment itself would be difficult. The centrally commissioned assessment would be more likely than an insurance-led one to need to assess the magnitude of biosecurity risks, which is particularly hard to model. However, stakeholders might have incentives to put pressure on regulators to make certain decisions. This is more likely to be successful because of the amount of judgment involved in the calculations of risk. Unlike the insurance-led solution, there would be no countervailing profit motive.
 Comparisons and Discussion The 2 approaches discussed would work by aligning the incentives for scientists and for funding bodies more closely with those of society as a whole. This approach keeps assessment of the benefits of scientific research purely in the hands of scientists, while bringing the right skills to bear at the right points and managing risk proportionately. We have explored 2 different ways to achieve this. Each has its advantages and disadvantages. The liability approach is more market based. As a result, the risk assessors have a financial incentive to accurately estimate risk, and political pressures are diminished. It might also be easier to use as a template internationally. Since the risks are global and the potentially risky research is not being pursued in just 1 country, being able to build global solutions is extremely valuable. The main benefit of the centrally commissioned analysis is it could bypass lack of clarity about who is liable for eventual damages in cases of biosecurity. These approaches have clear application for gain-offunction research of concern in the life sciences. However, similar approaches may work in a broader range of fields. The insurance mechanism is best suited for dual-use work where risk is predominantly through accident rather than
 { We are grateful to Piers Millett for this suggestion. BALANCING RISKS AND BENEFITS OF RESEARCH malicious intent. In all cases, pricing the absolute size of the risk and leaving researchers in charge of decisions about how to spend their budgets might offer a more responsive and proportionate approach to the challenges of funding research with potential public risks than existing systems. 			 Volume 15, Number 4, 2017


Abstract: Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as integral control; our terms introduce proportional and derivative control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Introduction Reinforcement learning has solved sequential decision tasks of impressive difficulty by maximizing reward functions through trial and error. Recent examples using deep learning range from robotic locomotion  (Schulman et al., 2015; Gu et al., 2016; Schulman et al., 2017; Levine et al., 2016)  to sophisticated video games  (Mnih et al., 2013; Schulman et al., 2017; OpenAI, 2018; Jaderberg et al., 2019) . While errors during training in these domains come without cost, in some learning scenarios it is important to limit the rates of hazardous outcomes. One example would be wear and tear 1 University of California, Berkeley 2 OpenAI. Correspondence to: Adam Stooke <<EMAIL>. Proceedings of the 37 th International Conference on Machine  Learning, Vienna, Austria, PMLR 119, 2020.  Copyright 2020 by the author(s). on a robot's components or its surroundings. It may not be possible to impose such limits by prescribing constraints in the action or state space directly; instead, hazard-avoiding behavior must be learned. For this purpose, we use the well-known framework of the constrained Markov decision process (CMDP)  (Altman, 1999) , which limits the accumulation of a "cost" signal which is analogous to the reward. The optimal policy is one which maximizes the usual return while satisfying the cost constraint. In safe RL the agent must avoid hazards not only at convergence, but also throughout exploration and learning. Lagrangian methods are a classic approach to solving constrained optimization problems. For example, the equalityconstrained problem over the real vector x: min x f (x) s.t. g(x) = 0 (1) is transformed into an unconstrained one by introduction of a dual variable-the Lagrange multiplier, λ-to form the Lagrangian: L(x, λ) = f (x) + λg(x), which is used to find the solution as: (x * , λ * ) = arg max λ min x L(x, λ) (2) Gradient-based algorithms iteratively update the primal and dual variables: −∇ x L(x, λ) = − ∇ x f (x) − λ∇ x g(x) (3) ∇ λ L(x, λ) = g(x) (4) so that λ acts as a learned penalty coefficient in the objective, leading eventually to a constraint-satisfying solution (see e.g.  Bertsekas (2014) ). The Lagrangian multiplier method is readily adapted to the constrained RL setting  (Altman, 1998; Geibel & Wysotzki, 2011)  and has become a popular baseline in deep RL  (Achiam et al., 2017; Chow et al., 2019)  for its simplicity and effectiveness. Although they have been shown to converge to optimal, constraint-satisfying policies  (Tessler et al., 2018; Paternain et al., 2019) , a shortcoming of gradient Lagrangian methods for safe RL is that intermediate iterates often violate constraints. Cost overshoot and oscillations are in fact inherent to the learning dynamics  (Platt & Barr, 1988; Wah et al., 2000) , and we witnessed numerous problematic cases in our own experiments. Figure  1  (left) shows an example from a deep RL setting, where the cost and multiplier values oscillated throughout training. Our key insight in relation to this deficiency is that the traditional Lagrange multiplier update in (4) amounts to integral control on the constraint. The 90-degree phase shift between the curves is characteristic of ill-tuned integral controllers. Our contribution is to expand the scope of possible Lagrange multiplier update rules beyond (4), by interpreting the overall learning algorithm as a dynamical system. Specifically, we employ the next simplest mechanisms, proportional and derivative control, to λ, by adding terms corresponding to derivatives of the constraint function into (4) (derivatives with respect to learning iteration). To our knowledge, this is the first time that an expanded update rule has been considered for a learned Lagrange multiplier. PID control is an appealing enhancement, evidenced by the fact that it is one of the most widely used and studied control techniques  ( Åström & Hägglund, 2006) . The result is a more responsive safety mechanism, as demonstrated in Figure  1  (right), where the cost oscillations have been damped, dramatically reducing violations. Our contributions in this paper are outlined as follows. First, we provide further context through related works and preliminary definitions. In Section 4, we propose modified Lagrangian multiplier methods and analyze their benefits in the learning dynamics. Next, in Section 5, we cast constrained RL as a dynamical system with the Lagrange multiplier as a control input, to which we apply PID control as a new algorithm. In Section 6, we adapt a leading deep RL algorithm, Proximal Policy Optimization (PPO)  (Schulman et al., 2017)  with our methods and achieve state of the art performance in the OpenAI Safety-Gym suite of environments  (Ray et al., 2019) . Finally, in Section 7 we introduce another novel technique that makes tuning easier by providing invariance to the relative numerical scales of rewards and costs, and we demonstrate it in a further set of experiments. Our extensive empirical results show that our algorithms, which are intuitive and simple to implement, improve cost performance and promote hyperparameter robustness in a deep RL setting.
 Related Work Constrained Deep RL. Adaptations of the Lagrange multiplier method to the actor-critic RL setting have been shown to converge to the optimal, constraint-satisfying solution under certain assumptions  (Tessler et al., 2018) . Convergence proofs have relied upon updating the multiplier more slowly than the policy parameters  (Tessler et al., 2018; Paternain et al., 2019) , implying many constraint-violating policy iterations may occur before the penalty comes into full effect. Several recent works have aimed at improving constraint satisfaction in RL over the Lagrangian method, but they tend to incur added complexity.  Achiam et al. (2017)  introduced Constrained Policy Optimization (CPO), a policy search algorithm with near-constraint satisfaction guarantees at every iteration, based on a new bound on the expected returns of two nearby policies. CPO includes a projection step on the policy parameters, which in practice requires a time-consuming backtracking line search. Yet, simple Lagrangian-based algorithms performed as well or better in a recent empirical comparison in Safety Gym  (Ray et al., 2019) . Approaches to safe RL based on Lyapunov functions have been developed in a series of studies  (Chow et al., 2018; , resulting in algorithms that combine a projection step, as in CPO, with action-layer interventions like the safety layer of  Dalal et al. (2018) . Experimentally, this line of work showed mixed performance gains over Lagrangian methods, at a nontrivial cost to implement and without clear guidance for tuning.  developed interior point methods for RL, which augment the objective with logarithmic barrier functions. These methods are shown theoretically to provide suboptimal solutions. Furthermore, they require tuning of the barrier strength and typically assume already feasible iterates, the latter point possibly being problematic for random agent initializations or under noisy cost estimates. Most recently,  Yang et al. (2020)  extended CPO with a two-step projection-based optimization approach. In contrast to these techniques, our method remains nearly as simple to implement and compute as the baseline Lagrangian method. Dynamical Systems View of Optimization. Several recent works have proposed different dynamical systems viewpoints to analyze optimization algorithms, including those often applied to deep learning.  Hu & Lessard (2017)  reinterpreted first-order gradient optimization as a dynamical system; they likened the gradient of the objective, ∇ x f , to the plant, which the controller aims to drive to zero to arrive at the optimal parameters, x * . Basic gradient de-scent then matches the form of integral control (on ∇ x f ). They extend the analogy to momentum-based methods, for example linking Nesterov momentum to PID control with lag compensation. In another example,  An et al. (2018)  interpreted SGD as P-control and momentum methods as PI-control. They introduced a derivative term, based on the change in the gradient, and applied their resulting PID controller to improve optimization of deep convolutional networks. Other recent works bring yet other perspectives from dynamical systems to deep learning and optimization, see for example  (Lessard et al., 2014; Nishihara et al., 2015; Liu & Theodorou, 2019) ). None of these works address constrained RL, however, necessitating our distinct formulation for that problem. Constrained Optimization. Decades' worth of literature have accumulated on Lagrangian methods. But even recent textbooks on the topic  (Bertsekas, 2014; Nocedal & Wright, 2006)  only consider updating the Lagrange multiplier using the value of the constraint function, g(x), and miss ever using its derivatives, ġ(x) or g(x), which we introduce. The modification to the Lagrangian method most similar in effect to our proportional control term (here using ġ(x)) is the quadratic penalty method  (Hestenes (1969) ;  Powell (1969)  see also e.g.  Bertsekas (1976) ), which we compare in Section 4.  Song & Leland (1998)  proposed a controls viewpoint (continuous-time) of optimizing neural networks for constrained problems and arrived at proportional control rules only. Related to our final experiments on reward-scale invariance,  Wah et al. (2000)  developed an adaptive weighting scheme for continuous-time Lagrangian objectives, but it is an intricate procedure which is not straightforwardly applied to safe RL.
 Preliminaries Constrained Reinforcement Learning Constrained Markov Decision Processes (CMDP)  (Altman, 1998)  extend MDPs (see  Sutton & Barto (1998) ) to incorporate constraints into reinforcement learning. A CMDP is the expanded tuple (S, A, R, T, µ, C 0 , C 1 , ..., d 0 , d 1 , ...), with the cost functions C i : S × A × S → R defined by the same form as the reward, and d i : R denoting limits on the costs. For ease of notation, we will only consider a single, all-encompassing cost. The expected sum of discounted rewards over trajectories, τ = (s 0 , a 0 , s 1 , a 1 , ...), induced by the policy π(a|s) is a common performance objective: J(π) = E τ ∼π [ ∞ t=0 γ t R(s t , a t , s t+1 )]. The analogous value function for cost is defined as: J C (π) = E τ ∼π [ ∞ t=0 γ t C(s t , a t , s t+1 )]. The constrained RL problem is to solve for the best feasible policy: π * = arg max π J(π) s.t. J C (π) ≤ d (5) Deep reinforcement learning uses a deep neural network for the policy, π θ = π(•|s; θ) with parameter vector θ, and policy gradient algorithms improve the policy iteratively by gathering experience in the task to estimate the reward objective gradient, ∇ θ J(π θ ). Thus our problem of interest is better expressed as maximizing score at some iterate, π k , while ideally obeying constraints at each iteration: max π J(π k ) s.t. J C (π m ) ≤ d m ∈ {0, 1, ..., k} (6) Practical settings often allow trading reward performance against some constraint violations (e.g. the constraints themselves may include a safety margin). For this purpose we introduce a constraint figure of merit with our experiments.
 Dynamical Systems and Optimal Control Dynamical systems are processes which can be subject to an external influence, or control. A general formulation for discrete-time systems with feedback control is: x k+1 =F (x k , u k ) y k =Z(x k ) u k =h(y 0 , ..., y k ) (7) with state vector x, dynamics function F , measurement outputs y, applied control u, and the subscript denoting the time step. The feedback rule h has access to past and present measurements. A problem in optimal control is to design a control rule, h, that results in a sequence y 0:T . = {y 0 , ..., y T } (or x 0:T directly) that scores well according to some cost function C. Examples include simply reaching a goal condition, C = |y T − y|, or following close to a desired trajectory, y 0:T . Systems with simpler dependence on the input are generally easier to analyze and control (i.e. simpler h performs well), even if the dependence on the state is complicated  (Skelton, 1988) . Control-affine systems are a broad class of dynamical systems which are especially amenable to analysis  (Isidori et al., 1995) . They take the form: F (x k , u k ) = f (x k ) + g(x k )u k (8) where f and g may be nonlinear in state, and are possibly uncertain, meaning unknown. We will seek control-affine form for ease of control and to support future analysis.
 Modified Lagrangian Methods for Constrained Optimization Lagrangian methods are a classic family of approaches to solving constrained optimization problems. We propose an intuitive, previously overlooked form for the multiplier update and derive its beneficial effect on the learning dynamics. We begin by reviewing a prior formulation for the equality-constrained problem. 1 4.1. Review: "Basic Differential Multiplier Method" We follow the development of  Platt & Barr (1988) , who analyzed the dynamics of a continuous-time neural learning system applied to this problem (our result can similarly be derived for iterative gradient methods). They begin with the component-wise differential equations: ẋi = − ∂L(x, λ) ∂x i = − ∂f ∂x i − λ ∂g ∂x i (9) λ = α ∂L(x, λ) ∂λ = αg(x) (10) where we have inserted the scalar constant α as a learning rate on λ. Differentiating (9) and substituting with (10) leads to the second-order dynamics, written in vector format: ẍ + A ẋ + αg(x)∇g = 0 (11) which is a forced oscillator with damping matrix equal to the weighted sum of Hessians: A ij = ∂ 2 f ∂x i ∂x j + λ ∂ 2 g ∂x i ∂x j , or, A = ∇ 2 f + λ∇ 2 g (12) Platt &  Barr (1988)  showed that if A is positive definite, then the system (11) converges to a solution that satisfies the constraint.  Platt & Barr (1988)  also noted that the system (9)-(  10 ) is prone to oscillations as it converges into the feasible region, with frequency and settling time depending on α. We provide complete derivations of the dynamics in (11) and for our upcoming methods in an appendix.
 Proportional-Integral Multiplier Method In (10), λ simply integrates the constraint. To improve the dynamics towards more rapid and stable satisfaction of constraints, we introduce a new term in λ that is proportional to the current constraint value. In the differential equation for λ, this term appears as the time-derivative of the constraint: λ = αg(x) + β ġ(x) = αg(x) + β j ∂g ∂x j ẋj ( 13 ) with strength coefficient, β. Replacing (10) by (  13 ) and combining with (9) yields similar second-order dynamics as (11), except with an additional term in the damping matrix: ẍ + A + β∇g∇ ⊤ g ẋ + αg(x)∇g = 0 (14) The new term is beneficial because it is positive semidefinite-being the outer product of a vector with itself-so it can increase the damping eignevalues, boosting convergence. The results of  (Platt & Barr, 1988 ) hold under  (13, 14) , because the conditions of the solution, namely ẋ = 0 and g(x) = 0, remain unaffected and extend immediately to ġ(x) = 0 (and for the sequel, to g(x) = 0). To our knowledge, this is the first time that a proportional-integral update rule has been considered for a learned Lagrange multiplier. The well-known penalty method  (Hestenes, 1969; Powell, 1969)  augments the Lagrangian with an additional term, c 2 g(x) 2 , which produces a similar effect on the damping matrix, as shown in  (Platt & Barr, 1988) : A penalty = A + c∇g∇ ⊤ g + cg(x)∇ 2 g (15) Our approach appears to provide the same benefit, without the following two complications of the penalty method. First, the penalty term must be implemented in the derivative ẋ, whereas our methods do not modify the Lagrangian nor the derivative in (9). Second, the penalty introduces another instance of the hessian ∇ 2 g in the damping matrix, which might not be positive semi-definite but shares the proportionality factor, c, with the desired term.
 Integral-Derivative Multiplier Method A similar analysis extends to the addition of a term in λ based on the derivative of the constraint value. It appears in λ as the second derivative of the constraint: λ = αg(x) + γg(x) (16) with strength coefficient γ. The resulting dynamics are: ẍ + B −1 A ẋ + αg(x) + γ ẋ⊤ ∇ 2 g ẋ B −1 ∇g = 0 (17) with B = I + γ∇g∇ ⊤ g , and I the identity matrix. The effects of the derivative update method are two-fold. First, since the eigenvalues of the matrix B −1 will be less than 1, both the damping (A) and forcing (∇g) terms are weakened (and rotated, generally). Second, the new forcing term can be interpreted as a drag quadratic in the speed and modulated by the curvature of the constraint along the direction of motion. To illustrate cases, if the curvature of g is positive along the direction of travel, then this term becomes a force for decreasing g. If at the same time g(x) > 0, then the traditional force will also be directed to decrease g, so the two will add. On the other hand, if g curves negatively along the velocity, then the new force promotes increasing g; if g(x) > 0, then the two forces subtract, weakening the acceleration ẍ. By using curvature, the derivative method acts predictively, but may be prone to instability. The proportional-integral-derivative multiplier method is the combination of the previous two developments, which induced independent changes in the dynamics (i.e. insert the damping matrix of (  14 ) into (  17 )). We leave for future work a more rigorous analysis of the effects of the new terms, along with theoretical considerations of the values of coefficients α, β, and γ. In the next section, we carry the intuitions from our analysis to make practical enhancements to Lagrangian-based constrained RL algorithms.
 Feedback Control for Constrained RL We advance the broader consideration of possible multiplier update rules by reinterpreting constrained RL as a dynamical system; the adaptive penalty coefficient is a control input, and the cost threshold is a setpoint which the system should maintain. As the agent learns for rewards, the upward pressure on costs from reward-learning can change, requiring dynamic response. In practical Lagrangian RL, the iterates λ k may deviate from the optimal value, even for lucky initialization λ 0 = λ * , as the policy is only partially optimized at each iteration. Adaptive sequences λ 0 , ..., λ K other than those prescribed by the Lagrangian method may achieve superior cost control for Problem (6). In this section we relate the Lagrangian method to a dynamical system, formalizing how to incorporate generic update rules using feedback. We return to the case of an inequality constrained CMDP to present our main algorithmic contribution-the use of PID control to adapt the penalty coefficient.
 Constrained RL as a Dynamical System We write constrained RL as the first-order dynamical system: θ k+1 =F (θ k , λ k ) y k =J C (π θ k ) λ k =h(y 0 , ..., y k , d) (18) where F is an unknown nonlinear function 2 corresponding to the RL algorithm policy update on the agent's parameter vector, θ. The cost-objective serves as the system measure, y, which is supplied to the feedback control rule, h, along with cost limit, d. From this general starting point, both the RL algorithm, F , and penalty coefficient update rule, h, can be tailored for solving Problem (6). The reward and cost policy gradients of the first-order 3 Lagrangian method, ∇ θ L(θ, λ) = ∇ θ J(π θ ) − λ∇ θ J C (π θ ), can be organized into the form of (18) as: F (θ k , λ k ) = f (θ k ) + g(θ k )λ k (19) f (θ k ) = θ k + η∇ θ J(π θ k ) (20) 2 Known as an "uncertain" nonlinear function in the control literature, meaning we lack an analytical expression for it.  3  We discuss only the first-order case, which provides sufficient clarity for our developments.
 g(θ k ) = −η∇ θ J C (π θ k ) (21) with SGD learning rate η. The role of the controller is to drive inequality constraint violations (J c − d) + to zero in the presence of drift from reward-learning due to f . The Lagrange multiplier update rule for an inequality constraint uses subgradient descent: λ k+1 = (λ k + K I (J C − d)) + (22) with learning rate K I and projection into λ ≥ 0. This update step is clearly an integral control rule, for h.
 Constraint-Controlled RL Our general procedure, constraint-controlled RL, is given in Algorithm 1. It follows the typical minibatch-RL scheme, and sampled estimates of the cost criterion, ĴC are fed back to control the Lagrange multiplier. In contrast to prior work  (Tessler et al., 2018; Paternain et al., 2019)  which uses a single value approximator and treats r + λc as the reward, we use separate value-and cost-value approximators, since λ may change rapidly. When λ is large, the update in (  19 ) can cause excessively large change in parameters, θ, destabilizing learning. To maintain consistent step size, we use a re-scaled objective for the θ-learning loop: θ * (λ) = arg max θ J − λJ C = arg max θ 1 1 + λ (J − λJ C ) This convex combination of objectives yields the policy gradient used in Algorithm 1. Our experiments use this re-scaling, including for traditional Lagrangian baselines.  a ∼ π(•|s; θ), s ′ ∼ T (s, a), 7: r ∼ R(s, a, s ′ ), c ∼ C(s, a, s ′ ) 8: Apply feedback control: Update critics, V φ (s), V C,ψ (s) ⊲ if using 13: ∇ θ L = 1 1+λ ∇ θ Ĵ(π θ ) − λ∇ θ ĴC (π θ ) 14: until converged 15: return π θ 16: end procedure As an aside, we note that it is possible to maintain the control-affine form of (  19 ) with this re-scaling, by reparam-eterizing the control as 0 ≤ u = λ 1+λ ≤ 1 and substituting for (21) with: g(θ k ) = −η∇ θ (J(π θ k ) + J C (π θ k )) (23) This parameterization simply weights the reward and cost gradients in the Lagrangian objective as: ∇ θ L(θ, λ) = (1 − u)∇ θ J(π θ ) − u∇ θ J C (π θ ) (24) It may provide superior performance in some cases, as it will behave differently in relation to the nonlinearity in control which arises from the inequality constraint. We leave experimentation with direct control on u ∈ [0, 1] to future work.
 The PID Lagrangian Method We now specify a new control rule for use in Algorithm 1. To overcome the shortcomings of integral-only control, we follow the developments of the previous section and introduce the next simplest components: proportional and derivative terms. Our PID update rule to replace (  22  return λ
 PID Control Experiments We investigated the performance of our algorithms on Problem (6) in a deep RL setting. In particular, we show the effectiveness of PID control at reducing constraint violations from oscillations and overshoot present in the baseline Lagrangian method. Both maximum performance and robustness to hyperparameter selection are considered. Although many methods exist for tuning PID parameters, we elected to do so manually, demonstrating ease of use.
 Environments: Safety-Gym We use the recent Safety-Gym suite  (Ray et al., 2019) , which consists of robot locomotion tasks built on the MuJoCo simulator  (Todorov et al., 2012) . The robots range in complexity from a simple Point robot to the 12-jointed Doggo, and they move in an open arena floor. Rewards have a small, dense component encouraging movement toward the goal, and a large, sparse component for achieving it. When a goal is achieved, a new goal location is randomly generated, and the episode continues until the time limit at 1,000 steps. Each task has multiple difficulty levels corresponding to density and type of hazards, which induce a cost when contacted by the robot (without necessarily hindering its movement). Hazards are placed randomly at each episode and often lay in the path to the goal. Hence the aims of achieving high rewards and low costs are in opposition. The robot senses the position of hazards and the goal through a coarse, LIDAR-like mode. The output of this sensor, along with internal readings like the joint positions and velocities, comprises the state fed to the agent. Figure  2  displays a scene from the DOGGOGOAL1 environment.
 Algorithm: Constraint-Controlled PPO We implemented Algorithm 1 on top of Proximal Policy Optimization (PPO)  (Schulman et al., 2017)  to make constraintcontrolled PPO (CPPO). CPPO uses an analogous clipped surrogate objective for the cost as for the reward. Our policy is a 2-layer MLP followed by an LSTM with a skip connection. We applied smoothing to proportional and derivative controls to accommodate noisy estimates. The environments' finite horizons allowed use of nondiscounted episodic costs as the constraint and input to the controller. Additional training details can be found in supplementary materials, and our implementation is available at https://github.com/astooke/safe-rlpyt.
 Main Results We compare PID controller performance against the Lagrangian baseline under a wide range of settings. Plots showing the performance of the unconstrained analogue confirm that constraints are not trivially satisfied, and they appear in supplementary material.
 ROBUST SAFETY WITH PI CONTROL We observed cost oscillations or overshoot with slow settling time in a majority of Safety Gym environments when using the Lagrangian method. Figure  3  shows an example where PI-control eliminated this behavior while maintaining good reward performance, in the challenging DOGGOBUTTON1 environment. Individual are plotted for different cost limits. As predicted in  (Platt & Barr, 1988) , we found the severity of cost overshoot and oscillations to depend on the penalty coefficient learning rate, K I . The top left panel of Figure  4  shows example cost curves from DOGGOGOAL2 under I-control, over a wide range of values for K I (we refer to varying K I , assuming K I = 1; the two are interchangeable in our design). With increasing K I , the period and ampli-tude of cost oscillations decrease and eventually disappear. The bottom left of Figure  4 , however, shows that larger K I also brings diminishing returns. We study this effect in the next section. The center and right columns of Figure  4  show the cost and return when using PI-control, with K P = 0.25 and K P = 1, respectively. Proportional control stabilized the cost, with most oscillations reduced to the noise floor for K I > 10 −4 . Yet returns remained relatively high over a wide range, K I < 10 −1 . Similar curves for other Safety Gym environments are included in an appendix.  We examine the trade-off between reward and constraint violation by forming an overall cost figure of merit (FOM). We use the sum of non-discounted constraint violations over the learning iterates, C F OM = k (D(π θ k ) − d) + , D(π θ ) = E τ ∼π T t=0 C(s t , a t , s ′ t ) , and estimate it online from the learning data. Figure  5  compares final returns against this cost FOM for the same set of experiments as in Figure  4 . Each point represents a different setting of K I , averaged over four runs. PI-control expanded the Pareto frontier of this trade-off into a new region of high rewards at relatively low cost which was inaccessible using the Lagrangian method. These results constitute a new state of the art over the benchmarks in  Ray et al. (2019) . We performed similar experiments on several Safety Gym environments in addition to DOGGOGOAL2: POINTGOAL1, the simplest domain with a point-like robot, CARBUTTON1, for slightly more challenging locomotive control, and DOG-GOBUTTON1 for another challenging task (see appendix for learning curves like Figure  4 ). for two strengths of added proportional control, for these environments. PI-control clearly improved the cost FOM (lower is better) for K I < 10 −1 , above which the fast integral control dominated. Hence robustness to the value for K I was significantly improved in all the learning tasks studied. . Learning run cost FOM versus penalty learning rate, KI , from four environments spanning the robots in Safety Gym. Each point is an average over four runs. In all cases, PI-control improves performance (lower is better) over a wide and useful range of KI , easing selection of that hyperparameter.
 CONTROL EFFICIENCY We further investigated why increasing the penalty learning rate, K I , eventually reduces reward performance, as was seen in the robustness study. Figure  7  shows learning curves for three settings: I-and PI-control with the same, moderate K I = 10 −3 , and I-control with high K I = 10 −1 . The high-K I setting achieved responsive cost performance but lower long-term returns, which appears to result from wildly fluctuating control. In contrast, PI-control held relatively steady, despite the noise, allowing the agent to do reward-learning at every iteration. The bottom panel displays individual control iterates, here displayed as u = λ/(1 + λ), over the first 7M environment steps, while the others show smoothed curves over the entire learning run, over 40M steps.  It was further able to slow the approach of the cost curve towards the limit, a desirable behavior for online learning systems requiring safety monitoring. Curves for other environments are available in an appendix.
 Reward-Scale Invariance In the preceding sections, we showed that PID control improves hyperparameter robustness in every constrained RL environment we tested. Here we propose a complementary method to promote robustness both within and across environments. Specifically, it addresses the sensitivity of learning dynamics to the relative numerical scale of reward and cost objectives. Consider two CMDPs that are identical except that in one the rewards are scaled by a constant factor, ρ. The optimal policy parameters, θ * remain unchanged, but clearly λ * must scale by ρ. To attain the same learning dynamics, all controller settings, λ 0 , K I , K P , and K D must therefore be scaled by ρ. This situation might feature naturally within a collection of related learning environments. Additionally, within the course of learning an individual CMDP, the balance between reward and cost magnitudes can change considerably, placing burden on the controller to track the necessary changes in the scale of λ. One way to promote performance of a single choice of controller settings across these cases would be to maintain a fixed meaning for the value of λ in terms of the relative influence of reward versus cost on the parameter update. To this end, we introduce an adjustable scaling factor, β k , in the policy gradient: ∇ θ L = (1 − u k )∇ θ J(π θ k ) − u k β k ∇ θ J C (π θ k ) (25) A conspicuous choice for β k is the ratio of un-scaled policy gradients: β ∇,k = ||∇ θ J(π θ k )|| ||∇ θ J C (π θ k )|| (26) since it balances the total gradient to have equal-magnitude contribution from reward-and cost-objectives at λ = 1 and encourages λ * = 1. Furthermore, β ∇ is easily computed with existing algorithm components. To test this method, we ran experiments on Safety Gym environments with their rewards scaled up or down by a factor of 10. Figure  9  shows a representative cross-section of results from the POINTGOAL1 environment using PI-control. The different curves within each plot correspond to different reward scaling. Without objective-scaling (i.e. β = 1), the dynamics under ρ = 10 are as if controller parameters were instead divided by 10, and likewise for ρ = 0.1. Note the near-logarithmic spacing of λ (λ ρ=10 has not converged to its full value). Using β ∇ , on the other hand, the learning dynamics are nearly identical across two orders of magnitude of reward scale. λ 0 = 1 becomes an obvious choice for initialization, a point where previous theory provides little guidance  (Chow et al., 2019)  (although here we left λ 0 = 0). Experiments in other environments and controller settings yielded similar results and are included in supplementary materials. Other methods, such running normalization of rewards and costs, could achieve similar effects and are worth investigating, but our simple technique is surprisingly effective and is not specific to RL.
 Conclusion Starting from a novel development in classic Lagrangian methods, we introduced a new set of constrained RL solutions which are straightforward to understand and implement, and we have shown them to be effective when paired with deep learning. Several opportunities for further work lay ahead. Analysis of the modified Lagrangian method and constrained RL as a dynamical system may relax theoretical requirements for a slowly-changing multiplier. The mature field of control theory (and practice) provides tools for tuning controller parameters. Lastly, the control-affine form may assist in both analysis (see  Liang-Liang Xie & Lei Guo (2000)  and  Galbraith & Vinter (2003)  for controllability properties for uncertain nonlinear dynamics) and by opening to further control techniques such as feedback linearization. Our contributions improve perhaps the most commonly used constrained RL algorithm, which is a workhorse baseline. We have addressed its primary shortcoming while preserving its simplicity and even making it easier to use-a compelling combination to assist in a wide range of applications. Figure 1 . 1 Figure1. Left: The traditional Lagrangian method exhibits oscillations with 90 • phase shift between the constraint function and the Lagrange multiplier, characteristic of integral control. Right: PID control on the Lagrange multiplier damps oscillations and obeys constraints. Environment: DOGGOBUTTON1, cost limit 200.
 ) is shown in Algorithm 2. The proportional term will hasten the response to constraint violations and dampen oscillations, as derived in Section 4. Unlike the Lagrangian update, derivative control can act in anticipation of violations. It can both prevent cost overshoot and limit the rate of cost increases within the feasible region, useful when monitoring a system for further safety interventions. Our derivative term is projected as (•) + so that it acts against increases in cost but does not impede decreases. Overall, PID control provides a much richer set of controllers while remaining nearly as simple to implement; setting K P = K D = 0 recovers the traditional Lagrangian method. The integral term remains necessary for eliminating steady-state violations at convergence. Our experiments mainly focus on the effects of proportional and derivative control of the Lagrange multiplier in constrained deep RL.Algorithm 2 PID-Controlled Lagrange Multiplier 1: Choose tuning parameters: K P , K I , K D ≥ 0 2: Integral: I ← 0 3: Previous Cost: J C,prev ← 0 4: repeat at each iteration k ← (K P ∆ + K I I + K D ∂) + 10: J C,prev ← J C 11:
 Figure 2 . 2 Figure 2. Rendering from the DOGGOGOAL1 environment from Safety Gym. The red, four-legged robot must walk to the green cylinder while avoiding other objects, and receives coarse egocentric sensor readings of their locations.
 Figure 3 . 3 Figure 3. Oscillations in episodic costs (and returns) from the Lagrangian method, KP = 0, KI = 10 −2 , are damped by proportional control, KP = 1 (ours), at cost limits 50, 100, 150, 200 (curves shaded) in DOGGOBUTTON1.
 Figure 4 . 4 Figure 4. Top row: Constraint-violating oscillations decrease in magnitude and period from increases in the Lagrange multiplier learning rate, KI . At all levels, oscillations are damped by PIcontrol, KP = 0.25, 1. Bottom row: Returns diminish for large KI ; proportional control maintains high returns while reducing constraint violations. Environment: DOGGOGOAL2, cost limit 50.
 Figure 5 . 5 Figure 5. Pareto frontier of return versus cost FOM, which improves (up and to the left) with PI-control, KP = 0.25, 1. Each point is a different setting of KI (see Figure 4).
 Figure6. Learning run cost FOM versus penalty learning rate, KI , from four environments spanning the robots in Safety Gym. Each point is an average over four runs. In all cases, PI-control improves performance (lower is better) over a wide and useful range of KI , easing selection of that hyperparameter.
 Figure 7 . 7 Figure 7. I-and PI-control with moderate KI = 10 −3 and Icontrol with fast KI = 10 −1 (IK I +). Top Returns diminished for fast-KI , but high for PI. Second Cost oscillations mostly damped by PI, removed by fast-KI . Third Control (smoothed) varies more rapidly under fast-KI , is relatively steady for PI. Bottom Control over first 500 RL iterations; fast-KI slams the control to the extremes, causing the diminished returns. Environment: DOG-GOBUTTON1, cost limit 200.
 Figure 8 . 8 Figure8. Derivative control can prevent cost overshoot and slow the rate of cost increase within feasible regions, which the Lagrangian method cannot do. Environment: DOGGOBUTTON1, cost limit 200.
 Figure 9 . 9 Figure 9. Costs, returns, and Lagrange multiplier with rewards scaled by ρ ∈ {0.1, 1, 10}; PI-control with KI = 1e − 3, KP = 0.1. Left column: without objective-weighting, learning dynamics differ dramatically due to required scale of λ. Right column: with objective-weighting, learning dynamics are nearly identical. Environment: POINTGOAL1, cost limit 25.
			 Standard techniques extend our results to inequality constraints, and multiple constraints, as in Platt & Barr (1988) , and notation is simplest for an equality constraint.