Lethal Autonomous Weapon Systems: Issues for Congress
Released: 2016-04-14T00:00:00

Introduction1

Many analysts and officials have indicated that this is a critical time in the research, development, and deployment of lethal autonomous weapon systems (LAWS), both in the United States and throughout the world. As discussed below, autonomous weaponry may play an increasingly important role in Department of Defense (DOD) plans for continued U.S. asymmetric advantage in combat. Such autonomy, however, also raises numerous concerns and some vocal opposition. These concerns are of three general types: (1) the belief that risks associated with such new weapons outweigh benefits, (2) concerns about whether lethal autonomy violates the international law of war, and (3) doubts regarding the moral impropriety of machines making apparently "discretionary" decisions to take a human life. Congress has an important role to play, either as part of the public discourse regarding the future of such capabilities and appropriate policy to address them, or "behind the scenes" via its funding authority and oversight responsibilities

Role of Congress

Questions related to the research on, and development and deployment of lethal autonomous weapon systems (LAWS) have been controversial for many years. However, several factors may call for congressional attention and potential action on LAWS at this time.

A nuanced understanding of LAWS and related issues may assist Congress in its role regulating the manning, funding, and equipping of U.S. military forces. LAWS are a component of the DOD's ongoing "third offset" strategy. This strategy is one way the DOD conceptualizes and integrates plans for ensuring continued asymmetric combat advantage for the United States, with particular focus on the incorporation of future technologies not easily replicated by competitor states or non-state entities. Robotics and autonomous systems have been highlighted by the DOD as a component of this overall future effort of the U.S. military.

Congress also sets the legal standards for the conduct of United States forces during armed conflict through the Uniform Code of Military Justice, as well as other statutory regulation. The use of LAWS involves many moral, ethical, and strategic issues beyond considerations of military advantage. For example, as discussed below, opponents of the development of LAWS argue, variously, that such weapon systems entail unrecognized long-term risks: strategic, such as undesirable escalation or difficulty maintaining control of the technology; legal, such as the inability of LAWs to discriminate between civilian and military targets; and ethical, because they place a machine in position to make a "discretionary" decision about human lives. The DOD currently internally regulates the research, development, and deployment of autonomous weapon systems via DOD Directive (DODD) 3000.09, Autonomy in Weapon Systems (2012). In the absence of congressional or executive action, some analysts consider this DOD directive as the de facto policy of the United States on this controversial topic.

Finally, Congress is an instrumental part of U.S. participation in internationally binding bodies and agreements, both via funding and treaty approval. There has been some recent consideration of LAWS at the United Nations via the Convention on Certain Conventional Weapons (CCW), the treaty that serves to restrict or ban internationally the use of certain weapons that are indiscriminant or that cause unnecessary suffering, which the United States ratified in 1995. States parties to the CCW and its various protocols agreed in 2013 to a mandate to review issues associated with LAWS, and convened meetings of experts in 2014 and 2015 to discuss these issues. Numerous international nongovernmental organizations (NGOs) view the CCW as the vehicle for advocating for multinational regulation or prohibition of LAWS. Senate approval would be required for any potential international treaty or additional protocol to the CCW, and congressional implementation may be required for any less formal agreement on the subject.

Options for Action

A variety of options have been proposed in response to the near-term development and possible appearance of LAWS in the battlespace. The option most often discussed is the proposal to enact a complete ban on research, development, and deployment of "fully" autonomous weapon systems. Proponents argue that such bans have been effective in the past—in areas such as biological and chemical weapons they have restricted use among major nation states and thereby retarded development—and could significantly curtail development and deployment, even among nations that do not voluntarily participate in the ban.

Opponents of the ban argue, in contrast, that a ban would be both undesirable and ineffective. They argue that it would be undesirable because of the substantial possibility that research in this area could eventually lead to the development of autonomous weapon systems that are more compliant with the law of armed conflict (LOAC) and other international law than current systems, as discussed below. It is also argued that such a ban would be ineffective because of two factors: (1) rapid development of civilian dual-use technologies, such as drone guidance systems and unmanned vehicles, and (2) non-U.S. peer development of these technologies. It is argued that peers will not agree to a ban, that such a ban will be unenforceable because of the ambiguity of such terms as "fully autonomous," and that such a ban, even if states publically agreed to it, would be unenforceable without a comprehensive and unlikely enforcement regime. Proponents of a ban have noted, though, that similar arguments have been raised with respect to bans on other technologies, such as blinding lasers, antipersonnel landmines, or cluster munitions, that have been negotiated and enforced.

Another option for action, advocated by both proponents and opponents of a ban, is regulation of the technology—both its development and deployment. Proponents of this idea suggest that autonomous weapon development should continue, but international bodies should develop regulatory guidelines, embodied in a binding agreement like a protocol to the CCW, to describe the appropriate contours for the use of autonomous systems. In the United States, the only regulatory document currently applying to autonomous weapon systems is the Department of Defense Directive 3000.09 – lauded by some in the international community for providing a model framework for testing and basic guiding principles. Others note that regulation at the international level can be coupled with transparency, particularly regarding LOAC compliance-testing methods and systems even if the autonomous source code remains secret, to ensure that developed systems are tested as vigorously and broadly as possible, minimizing the likelihood of unexpected decisions. In sum, in the absence of a complete ban, both opponents and supporters of a complete ban on lethal autonomous weapon systems agree that LAWS should be regulated and managed.

Congress would likely play a central role in any such ban or regulation of the technology, in a variety of ways. Of course, if a ban or control regime was developed via international treaty, then ratification of the treaty would require Senate approval. However, even in the absence of international action, Congress could set the legal bounds for the process of researching, developing, and deploying such systems within the DOD. Although the DOD Directive discussed above provides current standards for review and regulation of autonomous weapons, Congress could provide additional or alternate standards of review and employment.

Even if Congress does not seek to supplant the specific standards developed by the DOD in its directive, there are opportunities for congressional regulation and oversight. For example, although DODD 5000.01, Defense Acquisition Systems , requires legal review of weapon acquisitions, and there are pre-existing procedures for these reviews within each of the individual services, the weapons review process may not be adequate to handle the complexity, nuance, and transparency needed for autonomous weapon review. For example, as discussed below, an understanding and review of the nature and reliability of an autonomous system's behavior is required for adequate legal analysis. However, unlike traditional weapon reviews, lawyers making judgments about autonomous systems may require technical insight or even simulation capability currently unnecessary (and therefore unavailable) when evaluating more conventionally understood effects (such as explosive radius, etc.). To meet these challenges, weapons review in this and perhaps other areas of emerging technology may benefit from more detailed standardization and centralization, or the provision of additional resources. Furthermore, the complex issues and high international profile of these weapon systems might make it appropriate to require congressional reporting and thereby oversight at intermediate stages in the acquisition and legal review process not always necessary for other weapon systems.

Finally, Congress will almost certainly be presented with regulatory issues that relate to the development and employment of these technologies, but which do not directly relate to the standards of development and use within the United States. For example, Congress may be asked to consider statutory action to assist in LAWS development or prevent proliferation, perhaps by carefully regulating the export of dual use technologies in this area. Even in the absence of statutory regulatory action, congressional budgetary action on weapon system funding, as well as areas of research and development, will provide a direction for military development.

Defining Autonomous Weapon Systems

There are various ways to discuss autonomy in weapon systems. The definitions of the terms, and even the taxonomy of existing systems, are not always consistent among authors on the subject. As discussed in the text box, "What is Autonomy?" the synthesized view of the many definitions acknowledges a continuum of "autonomy" in weapon systems based primarily on two factors: (1) the target specificity (the geographic, temporal, and descriptive guidance designating the target of lethal force) provided by human operators when the weapon system is set into motion, and (2) the execution flexibility (scope of potential self-initiated action) in service to assigned goals.

Both the target specificity and execution flexibility of an autonomous system may vary by conflict, mission, or even individual objective. Therefore, a particular weapon system occupies a range rather than a point within the continuum of autonomy determined by its potential uses, and has a specific degree of autonomy only upon being set into motion with these parameters assigned. Discussions of the "autonomy" of a weapon system as a whole frequently refer to circumstances under which the system acts in a maximally autonomous manner. This

convention may be misleading and lead lawmakers and regulators to evaluate autonomy on a per platform basis, rather than defining permissible and impermissible conditions of employment that apply across devices.

The variety of military systems in use that automate some processes, or that include some degree of autonomy, is large, and a short survey may help to understand both their ubiquity and the scope of the systems' autonomy as perceived by various parties.

Force Multiplication

Military automation extends well beyond lethal autonomy to force-multiplication technologies, which are not explicitly considered in this report. This includes such disparate capabilities as automated drone flight (including takeoff and landing), auto-loitering capabilities of human-targeted weapons, and automated selection of high-interest imagery for intelligence analysis.

Even the new Joint Light Tactical Vehicle, a replacement for the Humvee, was planned to be manufactured by Oshkosh—a firm that offers software ("TerraMax") allowing their vehicles (including the model sold to the Army) self-driving capabilities. Autonomous systems of these types, which do not incorporate independent selection of targets or initiation of lethality, are not themselves controversial but nonetheless create both the technological and doctrinal basis for more hotly debated LAWS.

Defensive Systems

Another set of systems that incorporate some degree of autonomy along with lethality, but with less controversy, are autonomous defenses. The U.S. Navy, for example, has used the Phalanx system to defend ships against missile attack since the 1970s, with little comment from the civilian community. In cases where the ship defense systems recognize an incoming threat that requires a response faster than a human operator is capable of providing, the defense system is empowered to initiate a lethal response without human involvement. Likewise, a similar land-based system (C-RAM) has been deployed by the United States at forward-operating bases in Iraq. The C-RAM system, like the Israeli Iron Dome that performs a similar counter-rocket, artillery, and mortar function, can perform its defensive function only by detecting, targeting, and firing in a decision-cycle too fast for human operators to be involved.

In these defensive systems, the human operator does not designate a specific target and initiate the use of lethal force. While the absence of human control over targeting is often expressed as the break point for autonomous warfare, these systems are nonetheless frequently granted either a carve-out from otherwise restrictive regulations (as in the DODD), or treated as non-autonomous precursors to genuinely autonomous systems. This is likely related to both the high target specificity provided by the "defensive" nature of the weapons (targeting predetermined based on a specific set of geographic, temporal, and evaluative characteristics) and the relative lack of execution flexibility (these weapons simply shoot down objects that meet strictly defined criteria). These dual factors have led some, otherwise highly critical of autonomous weapon systems, to even decline to label them as autonomous—calling them "automated" instead.

Targeted Lethality

Another area of ubiquitous incorporation of some degree of autonomy is in weapon systems that exercise some degree of execution flexibility but have very high target specificity (see Figure 1 ). For these weapons, the specific individual target or group of targets (e.g., a specific plane or formation of planes, a specific structure) is designated by a person at the time of weapon initiation. This category includes, for example, cruise missiles, as well as the many air-to-air missiles that choose a target from among those available once launched into position by the human operator. Like defensive systems, these types of flexibly executing but very specifically targeted systems are generally considered to raise limited if any risk-based, legal, or moral/ethical issues associated with autonomy.

Other systems in this category, such as encapsulated torpedoes, have less specific targeting, and thereby have the potential to generate some controversy. An encapsulated torpedo is a stationary "mine" prepositioned in a guarded area that, when activated, targets and fires a torpedo at a hostile ship that enters the guarded area. In this case, the encapsulated torpedo shares some of the targeting specificity of defensive systems, but that specificity is reduced by its temporal separation from the human operator (it is prepositioned) and the potentially more nuanced and complex judgments required if the protected waterways are also used for civilian shipping. At the same time, these systems also incorporate the execution flexibility —via the torpedo's action—normally associated with systems featuring very specifically targeted lethality.

Autonomous Systems

A number of existing or proposed systems may already exhibit behavior that might be considered autonomous under generally prevailing standards. The Israeli Harpy system is an aerial drone that loiters in a target area, generally over enemy territory. Upon detecting a hostile radar source, the Harpy drone targets and initiates a lethal strike against that source. Because this system initiates lethal force against a target that has not been specifically designated by a human operator, it is plausibly considered an autonomous system by most definitions. Likewise, South Korea has deployed to the DMZ emplaced gun towers with autonomous lethal capacity, although their current operational assignment requires human consent before lethal response can be initiated.

Key Issues

A wide variety of topics are subject to debate in the policy and academic literature regarding the consideration and development of lethal autonomous weapon systems. Although an exact taxonomy does not exist, the numerous issues under debate can be usefully divided into those regarding (1) risks and potential benefits; (2) legal issues; and (3) moral/ethical concerns (see Figure 2 ). Although authors' positions vary in terms of nuance, much of the primary discussion centers on whether a ban (international or unilateral by the United States) on the research, development, and deployment of LAWS is appropriate.

Potential Benefits and Risks

Benefits

Capabilities in Military Context

That autonomous lethality provides tremendous potential value in the context of armed conflict is uncontroversial. With non-lethal military systems, traditional automation provides an immediate force-multiplier by taking repetitive or analytically arduous tasks and removing the need to hire, train, and support personnel to perform them. Autonomous action is more valuable, as complex systems that incorporate tools such as learning algorithms and contextual awareness allow for the "automation" of far more numerous and difficult (in terms of both training and incentive) tasks that require judgment and situational awareness. As a simple example, automation of some or all flight requirements of remote-controlled drones, if reliable, would allow for significant savings and multiplication of efforts by allowing remote-control pilots to assume direct control only during the actual operational use of the weapon system—automating the flight to and from the depot.

In addition, autonomous systems are generally capable of reacting substantially faster than humans. One way to conceptualize the critical element of initiative, as well as overall command and control competence, is the "OODA loop." The OODA loop consists of the key steps of (O)bserve, (O)rient, (D)ecide, and (A)ct. Under this concept, when considering two opposing forces, whether on the individual, tactical, or strategic level, whichever force has the ability to cycle through these steps the most quickly will control the initiative of the conflict—thereby forcing the opponent to react rather than initiate. In practice, this effect snowballs, as the faster force is able to counter-react before the opponent's initial reaction cycle completes, and each cycle of reaction delay drives the opponent more out of synch with appropriate response to the current situation. Some observers assert that the initial reaction advantage of autonomous systems will snowball into a potentially insurmountable advantage in warfare.

Finally, one of the primary concerns with today's non-autonomous remote controlled weapon systems is the problem of both unreliable connections to the remote pilot and the possibility of enemy interference. Like the presence of an on-board pilot, autonomous action by the weapon system itself minimizes the requirement for continuous communication and the possibility for enemy interference with control signals during deployment. Although the presence of software-driven decision-making raises the possibility of the enemy "hacking" autonomous control systems, it is unclear to what degree this risk is substantially greater than that already posed with modern non-autonomous weapon systems, almost all of which rely on sophisticated computer controls and, frequently, network communication.

However, these potential advantages are counterbalanced, even for many of those who do not support an outright ban, with concern for operational risks involved in LAWS development and deployment. While these risks are discussed in more detail below, they include the possibility that programming error, novel situations, or adversary activity could lead to a loss of control or predictability. Unlike idiosyncratic human decision-making, software control systems may be replicated across the fleet of LAWS, and so the damage potential of a simultaneous failure by all similar LAWS in the inventory must be considered, not only the consequences of a single system failure. This could result in disproportionately high damage versus human controlled or only partially autonomous, systems, with consequences including mass fratricide or undesired escalation of conflict.

Leverage Civilian Technology

Focus on lethal autonomous weapon systems may also be potentially beneficial for the United States because it capitalizes on current advances in civilian autonomous technology. The United States is a global leader in this area, and one of the imperatives of military technology is to maximize areas where an asymmetric advantage is available that is difficult for opponents to replicate. Synergistic technologies of stealth, reconnaissance, and precision weapons developed by the United States gave substantial and persistent advantage to the military precisely because these technologies represented areas of U.S. leadership and were difficult for opponents to replicate. Furthermore, investment by the United States in these areas of research and development will likely drive development of industrial capacity and commercial development in a virtuous cycle. Military and civilian developments in autonomous capability could therefore have a symbiotic relationship.

Translation of civilian developments in autonomy into weapon systems, however, may also result in an "arms race" dynamic, where competitor states are forced to invest in LAWS to retain military competitiveness; it could allow for the proliferation of lethal autonomy to entities, such as sub-state actors, who lack the organic R&D to otherwise develop such systems.

Potential Improvements in Ethical Warfare

Many authors, both opponents and supporters of a ban on LAWS, have highlighted the potential benefits of autonomous technology for ethical warfare in the sense that they could facilitate compliance with the law of armed conflict—at least in some areas. LAWS as currently conceived are not susceptible to emotional effects, such as shock or anger that may result in abuses by human soldiers. Finally, the presence of LAWs in mixed teams with human soldiers, particularly if LAWS have independent capacity to judge ethical conduct, may restrict the willingness and ability of those soldiers to engage in inappropriate or unlawful conduct.

In addition, introducing autonomous weapon systems into an environment where all or almost all of the potential targets are lawful, or have already been vetted, seems to potentially provide humanitarian benefits. For example, if the alternative is between introducing a lethal explosive device or a lethal autonomous system with some capability to avoid accidental or collateral casualties, the LAWS would likely be clearly legally and ethically desirable—even if the system's ability to distinguish non-combatants is unreliable. In this sense, autonomous decision-making at the moment of lethal action may be an improvement on the precision of weapon systems, eliminating some of the error created by imperfect intelligence and distance in time between the initiator and target.

However, these proposed benefits are questioned by many, including supporters of a ban, arguing that such "better ethical decision-making" technology does not exist and is unlikely to ever exist. There are also concerns that ethical decision-making would not be employed by potential state and non-state opponents of the United States in a prospective arms race, even if the United States reliably did it. The extensive legal and ethical critique of autonomous weapon systems arising from these questions is discussed in more detail below (under the " Legal Issues " and " Moral/Ethical Issues " sections).

Risks

Likelihood of War/Jus Ad Bellum

A common concern regarding the development of LAWS is that it will encourage inappropriate aggression. The justification for initiating armed conflict is generally described by the concept of jus ad bellum , or Just War theory. However, although sometimes couched as such, the concern that LAWS will lead to more warfare is not actually a legal one, since use of LAWS does not affect the legal evaluation of the propriety of war initiation. Rather, the argument is that LAWS would create a moral hazard for national leadership. This presupposes that current or future leaders are willing and desire to engage in unlawful war-making but are inhibited by the likelihood that it will result in military casualties, either for moral reasons or because of spin-off effects of those casualties. If these suppositions are accurate, then LAWS would appear to increase the likelihood that leaders would engage in unlawful aggression since it would minimize these casualties.

Some argue, however, that this objection seems excessively generic. They contend that any weapon system that minimizes casualties, or gives a substantial advantage to one side in armed conflict, would trigger this same moral hazard.

Uncontrolled Arms Race

Another potential risk to the development of LAWS that has been noted is that it will trigger wider arms races. This argument takes two forms. First, that because of the tremendous tactical advantage associated with the development of LAWS, peer and near-peer competitors will be forced to develop autonomous capabilities for their own weapon systems. Second, asymmetric competitors, such as international terrorist organizations, would have access to the technology once it becomes widely used in warfare. For both of these versions of an "arms race," one harm contemplated, in addition to the inherent instability associated with arms race dynamics , is that competitors will have either less incentive or less capacity to control the behavior of LAWS, resulting in development or fielding of LAWS that fail to comply with the laws of war (generally, this is conceived as competitors developing indiscriminate LAWS, since automation is far easier to accomplish than discrimination or ethical decision-making).

A number of counterpoints have been presented to this risk. First, many contend that an arms race is already in progress, with peer and near-peer competitors currently developing autonomous weapon systems—regardless of U.S. development of these systems. It is argued these nations would refuse to adopt, or successfully evade enforcement of, any potential multilateral ban. Second, it is argued that asymmetric competitors may be capable of taking advantage of technological development, particularly civilian sector advancements, even if not actively developed for military purposes by nation-states. Under this argument, once the basics of autonomy in machines are developed for civilian purposes, weaponization of these autonomous systems is relatively trivial.

Asymmetric Warfare

Another risk associated by some with the development of LAWS is an increased likelihood of attacks on civilian targets, particularly in the United States itself. The argument is that the development of LAWS will result in the absence of U.S. soldiers from the war zone. Enemies of the United States, it is argued, will see no political/strategic benefit in attempting to fight, or carry out attacks on autonomous weapon systems if the United States is not suffering human casualties. The opponent, under this argument, is therefore incentivized to carry out attacks on civilian rather than military targets.

Counter-arguments presented by others include at least one made against the discussion in the " Likelihood of War/ Jus Ad Bellum " section above, in that any generic technological advantage that makes U.S. service-members less susceptible to enemy attack appears to create the same risk. In the same vein, a DOD analyst has noted that this argument essentially "blames the victim," by discouraging protection of soldiers because of the enemy's presumed willingness to violate the laws of war by assaulting civilians. Finally, it has been pointed out, considering the history of nuclear strategy as well as terrorist targeting, that both peers and asymmetric opponents are not generally reluctant to place civilians in jeopardy if it serves strategic ends, and therefore the presence or absence of U.S. casualties away from the battlefield is irrelevant.

Hacking/Subversion

Another perceived risk with the use of autonomous weapon systems is that reliance on autonomous systems increases the military's vulnerability to hacking or subversion of software and hardware. The replication of software, as well as the complexity and interdependence involved with widespread use of autonomous weapon systems could also significantly magnify the harmful impact if a security vulnerability or exploitable system malfunction were discovered by an adversary. Potential consequences could include mass fratricide, civilian targeting, or unintended escalation (as discussed under " Loss of Command/Control " below). One response to that argument, however, is that "on-board" autonomous capability may counter subversion or hacking of current and future remote systems. Also, even weapon systems that do not include autonomous capabilities rely on computer hardware and software. This automation is no less susceptible to hacking and subversion, and the presence of autonomy may make a system more resilient than an equally computerized but less internally controlled non-autonomous weapon system.

Loss of Command/Control

Another risk discussed in the literature is the possibility that large-scale adoption of autonomous weapon systems may result in "run-away" escalation that results in warfare that otherwise would not have occurred. When considering this possibility, some of the military advantages of autonomous systems become disadvantages. First, the complexity, interdependence and flexibility of the system that allows it to perform complex mission sets may result in unpredictable and unintended lethality. In addition, some have maintained that the danger of uncontrolled escalation is significantly greater precisely because of the speed with which LAWS are capable of decision-making and action—one of the primary military advantages—creates a significant time delay between failure and corrective action. Some analysts of LAWS argue that in an environment with multiple autonomous systems—likely on both sides of a tense, armed confrontation—armed conflict may begin without either party intending it because of an initial error snowballing into a full-scale response, triggering an automated response in a vicious cycle.

The counter-argument is that there is nothing inherently more destructive about autonomous weaponry; it is simply conventional weaponry directed by an autonomous system. Because of this it is not clear why autonomous systems are more susceptible to inadvertent escalation than humans under the same circumstances. Some also question the plausibility of a scenario in which numerous free-ranging autonomous weapon systems come into contact with one another while empowered to engage in lethality independent of human tasking or authorization.

Judgment Errors/Accuracy

The final, and frequently primary, risk perceived by many is in the area of reliability and predictability. For various reasons, almost all involved in LAWS analysis recognize difficulties inherent in ensuring reliable decisionmaking. Proponents of a ban generally take the position

that the decision-making of an autonomous weapon system is fundamentally or irreducibly unpredictable, thereby foregoing the need for research to determine future reliability. For example, some argue that because no software can include an exhaustive description of all possible circumstances, it is impossible for an autonomous system to behave predictably outside highly controlled circumstances. Others argue that the technology required for flexible autonomous operations will, by needs, be based on learning or self-altering algorithms, which may develop unpredictable behavior patterns invisible to the original designers.

Some experts, however, believe that an autonomous decision-making system may plausibly reach a level of reliability and predictability comparable to a human soldier. The proponents of the technology, at least in theory, tend to argue that requiring absolute or logically certain predictability from LAWS holds it to a higher standard than that applied to humans and risks failing to use a potentially more reliable system because it is not perfectly reliable.

The question of decision-making performance is, however, inextricably linked to a large number of disputes regarding the legality of LAWS. The nature and performance of the autonomous system in making critical decisions about the propriety of the use of lethal force are the central issues of the next section.

Legal Issues124

The areas of legal contention regarding autonomous weapon systems are 1) the weapon system's ability to comply with U.S. obligations under international humanitarian law (IHL) and 2) rules of engagement. This is essentially an operational concern: "Will the functioning of the weapon systems comply with the appropriate requirements?" The second concern is less focused on function and more focused on accountability. This concern centers on whether the use of LAWS will make it more difficult to hold parties responsible for misconduct in the course of armed conflict.

Operational/Functional Laws

Various authors have pointed to three primary areas of operational law that may affect consideration of LAWS. First, there is the set of legal norms covered by the concept of jus ad bellum , which is the law governing the appropriate justification for the initiation of armed conflict. Second, there is the body of law classifying weapons as lawful or unlawful. Finally, all parties discuss the laws governing conduct during war, or jus in bello . Jus ad bellum is addressed in the section on " Likelihood of War/ Jus Ad Bellum " section above, because the relevant debate with respect to autonomous weapons has more to do with the perceived risk of moral hazard than legal justification for the use of force.

Weapons Law

A fundamental tenet of the international law of armed conflict is that "the right of the parties to an armed conflict to choose methods or means of warfare is not unlimited." Specifically, it is prohibited to use weapons or projectiles in such a manner as to cause superfluous injury or unnecessary suffering, or to use means or methods of warfare that are "intended to or may be expected to cause widespread, long-term, and severe damage to the natural environment." Under Article 36 of Additional Protocol I to the Geneva Conventions, states parties are also obligated to undertake legal reviews of new weapons systems under study, development, or acquisition, "to determine whether [their] employment would, in some or all circumstances, be prohibited by this Protocol or by any other rule of international law applicable to the High Contracting Party." While the United States is not a party to Additional Protocol I, it is one of the few states to have adopted a formal program to review weapons and weapon systems for compliance with international legal obligations.

A weapons evaluation for compliance with the laws of armed conflict considers first whether a weapon is prohibited per se , or prohibited under all circumstances, under the law of war. This status adheres to weapons that are banned pursuant to treaty as well as to weapons that cannot comply with legal requirements under any circumstance or method of use. The two principal legal requirements are, first, that the weapon does not cause suffering or injury beyond that required for a military purpose. For example, the use of glass ammunition is prohibited, without further evaluating the specific circumstances of use, because its use is considered to inflict unnecessary suffering. Second, weapons must be capable of being employed in a fashion to distinguish between military and civilian targets (which might be impossible because of an incapacity to target accurately or control effects). For example, a cyber-weapon that, when deployed, could not be prevented from doing uncontrollable collateral damage to civilian infrastructure would likely be illegal per se. Weapons are evaluated considering their normal or expected use rather than any conceivable use (or misuse).

Although some proponents of a ban on LAWS argue that such systems are per se illegal on the basis that they can never adequately distinguish between lawful and unlawful targets, opponents argue that this assertion ignores many lawful use scenarios. They point out that even "dumb" bombs are not per se illegal, since they can be used under circumstances in which civilians are not present; for example, to target a group of tanks in a desert area. Likewise, even autonomous weapons without any capability to distinguish between combatants and civilians might be used under limited circumstances in combat zones without noncombatants. The resolution of this disagreement seems to turn on the likelihood of any scenario in which LAWS can perform at least equal to a human, with opponents of a ban pointing to the uncontroversial current use of "over-the-horizon," or sensor-based, targeting as an analogy, and proponents of a ban arguing that these scenarios are extremely limited or unlikely.

The second aspect of a weapon evaluation is based on the specific proposed uses of the weapon. In this case, each of the proposed uses of the weapon must be evaluated for the weapon system's compliance—under those sets of circumstances—with the law of war. This contextual evaluation primarily relies on the weapon system's ability to comply with the principles of distinction and proportionality during actual operational use.

Law of Armed Conflict/Jus In Bello

Although a variety of "principles" form the basis of the law of armed conflict (the DOD identifies five), most authors considering autonomous weapon systems have centered their consideration on the foundational principle of distinction and its related principle of proportionality . The requirement to take feasible precautions is also frequently mentioned, but this issues seems to have generated little meaningful debate.

Distinction is the requirement that warring parties distinguish between military and civilian objects and personnel during the course of conflict, and is considered customary international law. As Article 48 of Additional Protocol I to the Geneva Conventions puts it, "[i]n order to ensure response for and protection of the civilian population and civilian objects, the Parties to the conflict shall at all times distinguish between the civilian population and combatants and between civilian objects and military objectives and accordingly shall direct their operations only against military objectives."

The primary concern, as discussed in the " Judgment Errors/Accuracy " section above, is that LAWS will simply be unable to distinguish between combatants and civilians. This inability is considered, by all sides of the debate, to be a particularly acute concern in the context of irregular warfare. In these conflicts, combatants may be embedded within the larger civilian environment, which creates extremely complex decision-making scenarios. As an example one author offers the case of an autonomous robot that performs a house-to-house search for combatants and encounters an individual running toward the robot, screaming, holding something metallic in his hand. One can certainly imagine circumstances in which entry into a civilian home would result in an agitated reaction from residents, and there are many objects that even humans are unable to quickly and effectively distinguish from weapons.

In addition, because LAWS lack empathy or human emotion, some authors argue that LAWS are now and will be in the future unable to effectively determine the intentions of individuals on the battlefield. As a result, LAWS will be unable to effectively distinguish between combatants and noncombatants, particularly, it is argued, in complex situations involving non-civilian noncombatants, such as surrendering, wounded, or otherwise incapacitated fighters.

Defenders of the technology, at least in terms of its potential, point out that future autonomous weapon systems may be more capable of distinguishing between combatants and civilians than human soldiers. LAWS' capabilities are not degraded by the same stress and emotional intensity that may affect the judgment of soldiers in combat. Moreover, because LAWS have no need for self-defense, they can respond more tolerantly to ambiguous circumstances than similarly situated soldiers, for example by delaying their response to "threatening" actions until the initiation of active hostility. In addition, governments interested in improving the accuracy of distinctions made by such systems could employ shared standards of testing, as well as leveraging the benefit of evaluation by ethicists of complex or difficult distinction decisions.

Others argue that LAWS will still be useful in high intensity conflicts, even if they never perform to a level permitting operation in combat zones that contain a significant number of noncombatants. For example, in a combat zone without noncombatants, a rule of engagement might allow any vehicle identified moving in an area of enemy encampment to be struck by a barrage of indirect fire from ship-based guns or "dumb" bombs dropped from the air. LAWS activity to target vehicles within this zone would have relatively low requirements to match human decision-making in similar circumstances. As long as LAWS are limited to these circumstances, their ability to perform extremely nuanced judgment tasks seems less relevant. Opponents counter that LAWS will inevitably be used outside these circumstances once available for operations because of the military advantages they provide. Whether or not this is true for U.S. military activities may turn on the criticality of the interest that the U.S. military force is protecting. It is clear that U.S. political and military leaders are willing to impose restrictions on military operations in many cases (e.g., Syria, Afghanistan); however, they may be less likely to maintain such restrictions if they believed the U.S. faced an existential threat. Analysts on both sides find the inappropriate use of LAWS by near-peer or non-state actors to be likely.

Proportionality

Proportionality is the requirement that military action not cause excessive damage to civilian lives or property in relation to the military advantage to be gained from the action. Articles 51 and 57 of Additional Protocol I to the Geneva Conventions prohibit attacks that "may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated."

Many argue that the proportionality judgment required by this rule is fundamentally beyond the capabilities of an autonomous system. "Military advantage" is perceived to be an inherently complex and flexible value, not susceptible to simulation by an autonomous system. When considering the allowable collateral impact of a single action (e.g., the dropping of a bomb), proportionality requires an understanding and integration of the surrounding circumstances of the immediate battlefield, as well as overall strategic understanding of the goals of the military action in question. The balance required in determining whether the collateral impact is "excessive" is argued to embed an inherently human judgment, as it relies upon the "reasonableness" of the determination. This "reasonableness" test, which forms so much of the basis for judging the legal propriety of human behavior, is a sort of rough-and-ready appeal to the human faculty of common sense and shared human values argued to be fundamentally inaccessible to LAWS.

Others who oppose a ban envision an autonomous weapon system in which the commander who set the LAWS in motion would make an initial judgment about whether accomplishing the mission goals programmed into the LAWS system was worth the expected collateral impact as a result of activation of the system. This judgment would include the established likelihood of unexpected action by the LAWS system. Once activated, LAWS operational evaluation of military advantage or allowable collateral impact levels could be determined in advance, requiring only a sensor judgment at the time of the assault to attempt to determine the amount of collateral impact, rather than setting the reasonable cut-off for aborting the action. While some critics have pointed out that such judgments are time-sensitive, and cannot simply be preprogrammed, others have responded that ensuring the reliability of these judgments simply requires setting time limitations as part of the mission framework for LAWS employment—so as to avoid the "aging" of the military advantage evaluation.

Opponents of a ban on LAWS have also pointed out that collateral damage estimates are regularly made using objective data and scientific algorithms with current weapon systems. It is also argued that many circumstances in modern warfare involve individuals executing the action (e.g., dropping the bomb, firing the missile) with little or no capability to assess the specific conditions of the target immediately prior to its destruction for an instantaneous proportionality assessment.

As noted above, the commander who sets the LAWS in motion plays a critical role in the legal responsibility for its resulting action. However, questions have been raised about whether that commander, or any other individual, could be held appropriately accountable for "war crimes" committed by such a weapon system. These concerns are further discussed below.

Accountability and Liability

Proponents of a ban on LAWS have raised a number of legal objections relating to the chain of accountability for the actions of these systems. Because machines are not ethical actors, proponents of a ban argue LAWS cannot meaningfully be "held responsible" for decision-making. As a result, if an autonomous system decided to carry out an action illegal under the laws of war (a "war crime"), holding someone responsible for that decision would be difficult or impossible.

Opponents of a ban counter that there is a long tradition of command responsibility for the actions taken by subordinates. They also point out that if the LAWS were intentionally designed or manufactured with the purpose of being used to commit war crimes, or with reasonable knowledge that they would be so employed, then the designers or manufacturers would have criminal liability. Likewise, if LAWS were used by a commander with the intention to commit a war crime, then the commander could likely be held responsible for that crime.

Proponents of the ban argue that war crimes are most likely to occur as a result of an unintended action by the autonomous system, not as an element of deliberate design. Although commanders are responsible for reasonably foreseeable actions of subordinates, these authors argue that commanders, designers, and manufacturers will be excused from such responsibility because of the fundamentally complex and unpredictable nature of autonomous decision-making. In this view, victims of war crimes committed by LAWS will lack redress, creating a fundamental lack of justice and responsibility associated with the weapons. For this reason alone, some argue, LAWS should be banned.

Opponents of the ban note that soldiers ordered to perform an otherwise lawful mission could commit war crimes as well. Ban proponents note that this still leaves someone criminally responsible for the misconduct, but opponents counter that this analysis places an excessive focus on individual criminal liability. They point out that the law has effectively managed responsibility for a variety of circumstances involving not fully predictable outcomes, such as the law regarding pet behavior or negligence. Moreover, the law of state responsibility would seem to allocate legal responsibility and an obligation to provide appropriate redress on the belligerent state employing the LAWS, arguably making the establishment of individual culpability less urgent.

The question of whether noncombatant victims of LAWS-related violence—whether collateral or accidental—can receive justice leads to a larger question about the moral propriety of LAWS.

Moral/Ethical Issues

The potential for autonomous weapon systems to make decisions about whether to take human life has generated discussion of risks and benefits, as well as legal concerns, but it has also raised more fundamental questions. Some, including Christopher Heyns (the United Nations, Human Rights Council Special Rapporteur on extrajudicial, summary, or arbitrary executions), have indicated that the very notion of machines making the decision to take a human life is morally problematic. As some describe, human dignity is at the core of the international law of human rights. They assert that allowing a machine to make an independent judgment to take a life negates that dignity. Others argue that allowing machines to make the decision to kill treats human being as objects, and denies their fundamental moral status.

Opponents of a ban argue that this moral intuition is based on excessive anthropomorphism of the autonomous weapon system, an analogy to human reasoning very unlikely to accurately reflect military technology within the foreseeable future. In their opinion, even a non-deterministic LAWS (e.g., using a flexible learning algorithm) is not making a "decision" in an ethically meaningful sense any more than is an air-to-air missile or patriot battery. Under this notion, the relevant decision to kill is made by the commander who assigns the LAWS mission, sets limits in time and space, describes Rules of Engagement, and sets the LAWS into motion. As discussed above, still other authors accept the LAWS as decision-maker in a morally relevant sense but argue that, when deployed, it will make better ethical decisions than a human soldier.

Appendix. Definitions of Autonomy

From HRW & IHRC, Shaking the Foundations, p. 1, "Fully autonomous weapons ... would identify and fire on targets without meaningful human intervention."

From Wallach and Allen, "Framing Robot Arms Control," p. 126, "Autonomous action by a robot includes any unsupervised activity."

From Anthony and Holland, "Governance of Autonomous Weapons," p. 424, "Contention issues centre on the weapon's adaptive capacity to make contingent discretionary decision and – in relation to those decisions – if, and at what point, a weapon is under human supervision."

From Heyns, Report of the Special Rapporteur, paragraph 38, "... robotic weapon systems that, once activated, can select and engage targets without further intervention by a human operator. The important element is that the robot has an autonomous 'choice' regarding selection of a target and the use of lethal force."

From Scharre and Horowitz, An Introduction to Autonomy in Weapon Systems, pp. 5-5, "What makes understanding autonomy so difficult is that autonomy can refer to at least three completely different concepts: * The human-machine command-and-control relationship * The complexity of the machine * The type of decision being automated."

From ICRC, Report of the ICRC Expert Meeting, p. 1, "There is no internationally agreed definition of autonomous weapon systems. For the purposes of this meeting, 'autonomous weapon systems' were defined as weapons that can independently select and attack targets, i.e. with autonomy in the 'critical functions' of acquiring, tracking, selecting and attacking targets."

From DODD 3000.09, Autonomy in Weapon Systems, p. 13, "A weapon system that, once activated, can select and engage targets without further intervention by a human operator. This includes human-supervised autonomous weapon systems that are designed to allow human operators to override operation of the weapon system, but can select and engage targets without further human input after activation."