Source: https://www.everycrsreport.com/reports/R45392.html
Timestamp: 2019-04-23 00:30:37+00:00

Document:
The nexus of robotics and autonomous systems (RAS) and artificial intelligence (AI) has the potential to change the nature of warfare. RAS offers the possibility of a wide range of platforms—not just weapon systems—that can perform “dull, dangerous, and dirty” tasks—potentially reducing the risks to soldiers and Marines and possibly resulting in a generation of less expensive ground systems. Other nations, notably peer competitors Russia and China, are aggressively pursuing RAS and AI for a variety of military uses, raising considerations about the U.S. military’s response—to include lethal autonomous weapons systems (LAWS)—that could be used against U.S. forces.
The adoption of RAS and AI by U.S. ground forces carries with it a number of possible implications, including potentially improved performance and reduced risk to soldiers and Marines; potential new force designs; better institutional support to combat forces; potential new operational concepts; and possible new models for recruiting and retaining soldiers and Marines. The Army and Marines have developed and are executing RAS and AI strategies that articulate near-, mid-, and long-term priorities. Both services have a number of RAS and AI efforts underway and are cooperating in a number of areas.
A fully manned, capable, and well-trained workforce is a key component of military readiness. The integration of RAS and AI into military units raises a number of personnel-related issues that may be of interest to Congress, including unit manning changes, recruiting and retention of those with advanced technical skills, training, and career paths.
RAS and AI are anticipated to be incorporated into a variety of military applications, ranging from logistics and maintenance, personnel management, intelligence, and planning to name but a few. In this regard, most consider it unlikely that appreciable legal and ethical objections to their use by the military will be raised. The most provocative question concerning the military application of RAS and AI being actively debated by academics, legal scholars, policymakers, and military officials is that of “killer robots” (i.e., should autonomous robotic weapon systems be permitted to take human life?).
Would an assessment of foreign military RAS and AI efforts and the potential impact on U.S. ground forces benefit policymakers?
Should the United States develop fully autonomous weapon systems for ground forces?
How will U.S. ground forces counter foreign RAS and AI capabilities?
How should the Department of Defense (DOD) and the Services engage with the private sector?
What are some of the personnel-related concerns associated with RAS and AI?
What role should Congress play in the legal and ethical debate on LAWS?
What role should the United States play in potential efforts to regulate LAWS?
New Models for Recruiting and Retaining Soldiers and Marines?
A Need for Systems and Tactics to Defend Against RAS and AI Use?
How Should DOD and the Services Engage with the Private Sector?
What Are Potential Personnel-Related Concerns Associated with RAS and AI?
The nexus of robotics and autonomous systems (RAS) and artificial intelligence (AI) has the potential to change the nature of warfare. RAS offers the possibility of a wide range of platforms—not just weapon systems—that can perform "dull, dangerous, and dirty" tasks—potentially reducing the risks to soldiers and Marines and possibly resulting in a generation of less expensive ground systems. Other nations, notably peer competitors Russia and China, are aggressively pursuing RAS and AI for a variety of military uses, raising considerations about the U.S. military's response—to include lethal autonomous weapons systems (LAWS)—that could be used against U.S. forces.
RAS and AI are anticipated to be incorporated into a variety of military applications, ranging from logistics and maintenance, personnel management, intelligence, and planning to name but a few. In this regard, most consider it unlikely that appreciable legal and ethical objections to their use by the military will be raised. The most provocative question concerning the military application of RAS and AI being actively debated by academics, legal scholars, policymakers, and military officials is that of "killer robots" (i.e., should autonomous robotic weapon systems be permitted to take human life?).
Apart from the U.S. military's pursuit of RAS and AI, there has been a proliferation of RAS and AI internationally, ranging from foreign militaries, to violent nonstate groups to criminal organizations. These technologies have already been used to a limited degree against U.S. military forces. How the United States will address further foreign advances in these realms may be of interest to Congress.
(1) Any artificial system that performs tasks under varying and unpredictable circumstances without significant human oversight, or that can learn from experience and improve performance when exposed to data sets.
(2) An artificial system developed in computer software, physical hardware, or other context that solves tasks requiring human-like perception, cognition, planning, learning, communication, or physical action.
(3) An artificial system designed to think or act like a human, including cognitive architectures and neural networks.
(4) A set of techniques, including machine learning that is designed to approximate a cognitive task.
(5) An artificial system designed to act rationally, including an intelligent software agent or embodied robot that achieves goals using perception, planning, reasoning, learning, communicating, decision-making, and acting.
Modern information technologies and the advent of machines powered by artificial intelligence (AI) have already strongly influenced the world of work in the 21st century. Computers, algorithms and software simplify everyday tasks, and it is impossible to imagine how most of our life could be managed without them. However, is it also impossible to imagine how most process steps could be managed without human force? The information economy characterized by exponential growth replaces the mass production industry based on economy of scales.
When we transfer the experience of the past to the future, disturbing questions arise: what will the future world of work look like and how long will it take to get there? Will the future world of work be a world where humans spend less time earning their livelihood? Alternatively, are mass unemployment, mass poverty and social distortions also a possible scenario for the new world, a world where robots, intelligent systems and algorithms play an increasingly central role? What is the future role of a legal framework that is mainly based on a 20th century industry setting? What is already clear and certain is that new technical developments will have a fundamental impact on the global labor market within the next few years, not just on industrial jobs but on the core of human tasks in the service sector that are considered 'untouchable.' Economic structures, working relationships, job profiles and well-established working time and remuneration models will undergo major changes.
Well-known examples from the field of robotics and AI are the so-called 'smart factories', driverless cars, delivery drones or 3D printers, which, based on an individual template, can produce highly complex things without changes in the production process or human action in any form being necessary.
If, as some suggest, society is in a "Fourth Industrial Revolution" what are the implications for the U.S. military as a whole and, in particular, U.S. ground forces, namely the Army and Marine Corps?
As Secretary of the Army Mark Esper reportedly noted, "Whoever gets to robotics and AI first, it'll be a game changer on the battlefield."22 In this regard, the stage appears set for nations to aggressively pursue RAS and AI over the near- and long-term to achieve a battlefield advantage.
RAS and AI have been described as changing the very nature of work and workforce design, with some experts predicting an acceleration of this trend over the next two decades.23 RAS and AI advances in the private sector in areas such as transportation, logistics, manufacturing, health care, and engineering could be readily adapted by the military and ground forces. A potential added incentive is the adoption of these technologies would likely face little international opposition, as these types of technologies do not readily fall into the category of autonomous weapons.
Will AI and RAS displace certain skills/tasks or entire jobs, and what types of roles will the new technologies fill? If certain skills/tasks are automated, employees might have opportunities to upgrade the skills of their employers. If entire jobs are eliminated, employers could be reluctant to maintain the size of their workforce and pay for employee re-skilling while also investing in AI and RAS technologies.
How will the pace of innovation and adoption of new technologies affect the workforce and corresponding labor policies? Some experts argue that AI and RAS technologies are developing much more rapidly than disruptive technologies of prior years (which have largely involved physical systems such as automated teller machines at banks) and a subsequent long time lag between innovation and full adoption. While there have been concerns about automation and new technologies displacing middle class workers for the past two centuries,26 the ability to implement AI systems and software on existing physical hardware could facilitate rapid adoption and allow for more disruptive changes to the labor market than have been seen historically. Further, RAS and AI technologies together can replace both physical and cognitive labor. Some analysts have raised concerns that wide spread adoption of AI and RAS systems might cause shifts in the workforce that outpace changes to labor policies.
Are there a sufficient number of AI and RAS experts in the workforce to implement the technologies across the public and private sectors? A number of studies have noted that there is far more demand than supply of such experts.27 What are the roles and responsibilities of the public and private sectors in meeting the demand?
Fewer soldiers and Marines could have a direct impact on the size and allocation of the defense budget, not just in military compensation, but also in military logistics, construction, and health care, for example.
Automated systems and AI can be of significant value in assisting military analysts, planners, and commanders in processing and synthesizing large and diverse data sets.
Automated systems and AI can also be of great value in dealing with military situations where a "manual" or human-in-the-loop response is insufficient, such as dealing with "swarms" of unmanned ground or aerial vehicles or an inbound hypersonic weapon.
RAS and AI have the potential to improve both the individual performance of troops as well as the performance of virtually every type of unit. RAS has applicability in lightening soldiers' and Marines' individual combat loads, improving situational awareness at the squad and platoon levels, and serving as "teammates" rather than simply tools.34 AI can be employed as a planning and decision support tool and could be a central component in automated weapon systems (AWS), which can provide protection from incoming aircraft, missiles, rockets, artillery and mortar shells, and other threats.
Some believe RAS and AI also show great promise in reducing physical risks to soldiers and Marines. RAS and AI can be used in such missions as explosive ordnance disposal (EOD), route clearance, obstacle breaching, and chemical, biological, radiological, and nuclear (CBRN) reconnaissance—all considered extremely high-risk operations—in a manner that significantly limits troop exposure to these hazards. As previously noted, RAS and AI are expected to play a greater role in force protection, particularly from the risk of aircraft, missile, rocket, and artillery and mortar attack.
In some cases, RAS- and AI-inspired force redesigns could not only be revolutionary but controversial as well. The Army, for example, plans to replace the current M-2 Bradley-series infantry fighting vehicle with an Optionally-Manned Fighting Vehicle (OMFV), which can also be operated remotely instead of by a crew.36 Remotely transporting an infantry unit in an OMFV could give rise to concerns that should the remote control capability fail or be disrupted, the vehicle's occupants would be unduly vulnerable to enemy fire.
From an institutional perspective, RAS and AI have a wide range of applicability in training and educating troops and leaders which, in addition to improved efficiency, could result in both cost savings as well freeing up personnel previously dedicated to these tasks for other assignments. From an institutional support perspective, RAS and AI developed for private sector use can most likely be readily adapted for military use. Military warehouse and depot functions are likely prime candidates for RAS and AI applications, potentially increasing efficiency, reducing costs, and freeing up personnel for other endeavors. Efforts are also underway for the partial automation of both ground and air logistics so that unmanned systems can be used to deliver supplies and evacuate casualties. Another potential application being explored by the Army is using AI to predict when vehicle parts might breakdown and prevent equipment failures before they happen.37 RAS and AI also have potential applications in treatment of wounded soldiers in combat zones and in rear areas as well.
While it remains to be seen to what extent the Army and Marines adopt RAS and AI for battlefield use, it is reasonable to assume potential adversaries might seek to pursue these capabilities, possibly even to a greater extent than the United States. For example, while the United States may choose not to pursue autonomous weapons based on moral, ethical, and legal considerations, other nations might not feel so obligated and aggressively develop these capabilities for possible use against U.S. forces. In this and other cases, the Army and Marines could be required to develop new systems, tactics, operational concepts, and possibly even units to counter the threat posed by enemy RAS and AI.
The following sections provide a brief illustrative description of selected non-U.S. RAS and AI efforts. For the selected countries, the efforts discussed are only examples and might not constitute that nation's entire military RAS/AI program.
The Vikhr is reportedly armed with a stabilized 30mm Shipunov 2A72 automatic cannon, a coaxial 7.62mm Kalashnikov PKT/PKTM machine gun, and six ready-to-launch 9M133M Kornet-M (AT-14 Spriggan) anti-tank guided missiles (ATGMs).49 The Vikhr can be reconfigured to accommodate a variety of weapons. The 2A72 main gun can be replaced by a single or twin-barrel 23mm 2A14 anti-aircraft cannon, 12.7mm NSVT or Kord heavy machine gun, or a 30mm Gsh-6-30K six-barrel naval automatic cannon. The Vikhr can also accommodate surface-to-air missiles of Igla (SA-18 Grouse) or 9K333 Verba man-portable air defense systems, as well as Shmel-M reactive flame throwers. Foreign artillery systems can also be integrated onto the Vikhr. The Vikhr can also be equipped with four mini unmanned aerial vehicles (UAVs) to provide a surveillance capability. The Vikhr is said to be remotely controlled up to a distance of 10 kilometers.
Source: Nikolai Novichkov, "New Russian Combat UGV Breaks Cover, Uran-9 Readies for Service," IHS Jane's International Defence Review, October 2016, p. 30.
People's Liberation Army PLA thinkers expect AI to reshape the character of war itself, from today's "informatized" ways of warfare into "intelligentized" warfare, in which AI is critical. According to Lt. Gen. Liu Guozhi, who leads the Central Military Commission's (CMC) Science and Technology Commission, AI could accelerate military transformation, reshaping military units' programming, operational styles, equipment systems, and models of combat power generation, ultimately leading to a profound military revolution.
Another report suggests China has already developed a range of unmanned aerial, underwater, surface, and ground platforms and is working on cutting-edge unmanned systems, including those with stealth, swarming, and hypersonic capabilities.54 China believes these modern unmanned systems could be used to introduce a persistent presence in disputed waters or territories.55 These reports suggest China has wide-ranging military applications for RAS and AI in mind, which could present a multifaceted challenge to the U.S. military.
The South Koreans have developed the Samsung SGR-A1 robot sentry to defend South Korea against North Korean border intrusion.58 In 2007, it was revealed the SGR-A1 had a fully autonomous mode, and a number of press sources began referring to it as a fully autonomous weapons system, which resulted in a great deal of negative press, although Samsung and South Korean officials noted that a human was required to engage targets. Reportedly, the SGR-A1s, which cost $200,000 apiece, are remotely operated sentries that mount either a 5.5mm machine gun or a 40mm automatic grenade launcher, which work in conjunction with cameras and radar systems that can detect intruders with heat and motion sensors and can challenge them through audio or video communications.59 The SGR-A1 is deployed throughout the 160-mile Korean demilitarized zone (DMZ).
The Brimstone missile developed for the Royal Air Force is an aircraft-launched, fire-and-forget missile designed to destroy ground vehicles or small boats (Figure 3).60 Brimstone has two modes of operation, one that involves a human "painting" a target with a laser and the other "fire and forget" mode where the missile's software seeks a predesignated target type within an established kill box. Brimstone has reportedly been used against Islamic State targets in Syria. Saudi Arabia has acquired the Brimstone missile as well.61 While the United States has a similar system—the Long Range Anti-Ship Missile (LRASM)—Brimstone is an example of an exportable, semi-autonomous weapon system (which could be converted to an autonomous weapon system by adding a loiter capability and switching it to a single fire-and-forget mode) against which U.S. ground forces will likely need to develop countermeasures.
Army and Marine RAS and AI efforts are governed by various policies and managed by a number of different organizations. The following sections provide an overview of selected authorities and managing organizations.
In addition to autonomous systems, it also addresses cyber operations, information assurance, the electromagnetic spectrum, and electronic warfare.
The Army has a variety of organizations involved in its RAS and AI efforts. In managing these efforts, there are three major organizations involved. At the Army level, the Assistant Secretary of the Army for Acquisitions, Logistics, and Technology (ASA [ALT]) Program Director for Robotics is responsible for RAS. In terms of major command-level management, the Training and Doctrine Command (TRADOC) Capability Manager for RAS exercises management of Army ground and air RAS efforts, as well as overall RAS integration. In July 2018, the Army stood up Army Futures Command (AFC),72 intended to establish unity of command and effort that consolidates the Army's modernization process under one roof. It is not yet established how AFC will manage the Army's RAS and AI efforts, but it is expected that AFC will have a major role in managing these efforts.
On October 2, 2018, the Army announced the creation of the Army Artificial Intelligence Task Force in Support of the Department of Defense Joint Artificial Intelligence Center (A-AI TF). The Army's intent is to "establish a scalable A-AI TF under U.S. Army Futures Command (AFC) consisting of hand-selected Army personnel with specific skill sets to lead Army AI efforts and support DOD projects, principally based at Carnegie Mellon University."74 The A-AI TF will work with Carnegie Mellon's National Robotics Engineering Center in Pittsburg, PA, and the Army expects to achieve an initial operating capability at Carnegie Mellon University in early November 2018.
In a similar manner, a number of organizations are involved in Marine RAS and AI efforts. The Marine Corps Combat Development and Integration Command (CDIC) is responsible for developing RAS and AI concepts of operation and employment. Next, the Marine Corps Warfighting Laboratory is responsible for scientific and technological (S&T) development of RAS and AI. Finally, the Marine Corps Systems Command (MCSC) is responsible for the acquisition of RAS and AI technologies/systems.
1. Increase situational awareness. Complex terrain and enemy countermeasures limit soldiers' abilities to see and fight at the battalion level and below. Advancements in RAS allow for persistent surveillance and reconnaissance over wide areas, often going where manned systems cannot, thereby increasing standoff distances, survivability and reaction time for commanders.
2. Lighten the soldiers' physical and cognitive workloads. Excessive equipment requirements reduce stamina and endurance. Autonomous systems lighten equipment loads and increase soldier speed, mobility, stamina and effectiveness. Vast amounts of information overload leaders' ability to make decisions. RAS facilitate mission command by collecting, organizing, and prioritizing data to facilitate decision-making as well as improving tactical mobility while reducing cyber, electronic, and physical signatures.
3. Sustain the force with increased distribution, throughput, and efficiency. Logistics distribution is resource intensive. Soldiers and teams become vulnerable at the end of extended supply lines. Air and ground unmanned systems and autonomy-based capabilities enhance logistics at every stage of supply movement to the most forward tactical resupply points. RAS move materiel to the most urgent points of need and provide options for Army logistics distribution to the warfighter.
4. Facilitate movement and maneuver. Joint combined arms maneuver in the 21st century requires ready ground combat forces capable of outmaneuvering adversaries physically and cognitively in all domains. Through credible forward presence and resilient battle formations, future ground forces integrate and synchronize joint, interorganizational, and multinational capabilities to create temporary windows of superiority across multiple domains; seize, retain, and exploit the initiative; and achieve military objectives. Investments in Anti-Access/Area Denial (A2/AD) capabilities allows future enemies to engage Army forces earlier and at greater distances. In addition, adversaries will look to emplace obstacles to threaten movement and maneuver across extended avenues of advance. As a counter, Army forces employ RAS to extend the depth of the area of operations and to provide responses to enemy action. RAS expand the time and space at which Army forces can operate and improve the ability to overcome obstacles.
Within the context of the Army's RAS Strategy, Army leadership has established near-, mid-, and far-term RAS priorities.
In a manner similar to the Army's, the Marines have established near-, mid-, and far-term priorities.
Increase situational awareness with advanced, smaller and swarming RAS.
Lighten the load with exoskeleton capabilities.
Improve sustainment with fully automated convoy operations.
Improve maneuver with unmanned combat vehicles and advanced payloads.
Enable manned and unmanned teaming (MUM-T).
Scalable sensors, scalable teaming to support MUM-T.
Autonomous unmanned aerial vehicle (UAV) navigation, for example, is relatively straightforward, since the world model according to which it operates consists simply of maps that indicate preferred routes, height obstacles and no-fly zones. Radars augment this model in real time by indicating which altitudes are clear of obstacles. Global Positioning System (GPS) coordinates convey to the UAV where it needs to go, with the overarching goal of the GPS coordinate plan being not to take the aircraft into a no-fly zone or cause it to collide with an obstacle.
In comparison, navigation for driverless cars is much more difficult. Cars not only need similar mapping abilities, but they must also understand where all nearby vehicles, pedestrians and cyclists are, and where all these are going in the next few seconds. Driverless cars (and some drones) do this through a combination of sensors like LIDAR (Light Detection and Ranging), traditional radars, and stereoscopic computer vision. Thus the world model of a driverless car is much more advanced than that of a typical UAV, reflecting the complexity of the operating environment. A driverless car computer is required to track all the dynamics of all nearby vehicles and obstacles, constantly compute all possible points of intersection, and then estimate how it thinks traffic is going to behave in order to make a decision to act.
Overcoming the challenges of tactical cross-country autonomous navigation (e.g., avoiding obstacles such as barbed wire, minefields, and antitank ditches) and using terrain to shield military vehicles from detection and engagement by direct-fire weapons remains, for the foreseeable future, a crucial developmental challenge for both U.S. and foreign ground forces.
The following sections provide a brief description of selected unclassified Army and Marine Corps RAS and AI efforts.
Source: Todd South, "These Two Army BCTs Will Be the First to Put Robotic Vehicles in Their Formations," Army Times, April 10, 2018.
The SMET is an unmanned robotic vehicle intended to provide logistical support to squads in Army Infantry Brigade Combat Teams (IBCTs) and Marine Infantry Battalions.87 The SMET will be designed to operate in unmanned and optionally manned modes and will be required to carry up to 1,000 lb., operate over 60 miles in 72 hours, and generate 3 kW stationary and 1 kW moving to enable equipment and charge batteries. The target cost for the SMET is no more than $100,000 per system. The SMET is largely a commercial off-the-shelf effort, as a number of vendors had previously developed prototypes for past Army robotic initiatives.
Conceptually, the SMET is intended to carry troops, food and water, ammunition, supplies, and other weapons such as mortars and anti-armor weapons. The SMET is also to be expandable and could conduct route clearance and breaching operations by the addition of special mission modules. Theoretically, the SMET could also be configured to conduct reconnaissance and serve as a semi- or fully autonomous weapon system when armed.
In December 2017, the Army selected four vendors—Team Polaris (Applied Research Associates, Polaris Defense, and Neya Systems), General Dynamics Land Systems, HDT Global, and Howe and Howe Technologies—to conduct a six-month operational technology demonstration in FY2019.88 Sixty-four SMETs (16 from each vendor) are to be made available to the 1st Brigade Combat Team (BCT), 10th Mountain Division, Ft. Drum, NY, and the 2nd BCT, 101st Airborne Division, Ft. Campbell, KY, for evaluation in November 2018. Army plans call for SMET to transition to a Program of Record by first quarter FY2020, after which the Army would transition into Low Rate Initial Production (LRIP) of the SMET. By the second or third quarter of FY2021, the first Army units will begin to receive the SMETs. Depending on budgets, the Army could eventually procure as many as 5,723 SMETs.
Source: Kevin Lilley, "Driverless Army Convoys: 6 Takeaways from the Latest Test," Army Times, July 1, 2016.
The Army's Leader-Follower Technology for Tactical Wheeled Vehicles (TWVs) effort revolves around a suite of sensors and vehicle upgrades intended to provide TWVs the capability of linking three unmanned vehicles to a single manned vehicle during the conduct of logistics road convoy operations.89 This effort is intended to reduce the number of soldiers required to operate a convoy, thereby reducing the number of exposed soldiers to risk of injury from attack.
The Army's Tank Automotive Research, Development and Engineering Center (TARDEC) of Warren, MI, is reportedly working with several industry partners on the effort.90 Robotic Research LLC, of Gaithersburg, MD, is providing the autonomy kit for the vehicles, and Oshkosh Defense of Oshkosh, WI, is building the kits that allow the trucks to be remotely operated. Lockheed Martin of Bethesda, MD, is the integrated systems developer, and DCS Corporation of Alexandria, VA, is creating the graphic user interface allowing the soldier in the lead vehicle to control the follower trucks.
Source: Army Briefing, Army Robotic and Autonomous Systems (RAS) Portfolio, April 11, 2018.
Source: Maddy Longwell, "The Army Hopes AI Can Help Prevent Vehicle Failure," C4ISRNet, June 26, 2018.
Source: Marine Corps Robotic and Autonomy Strategy (MCRAS), June 2018.
The Marine RV (M) is a custom-built, multimission-tracked platform that incorporates AI but also fulfils the need for man-in-the-loop capabilities. It is intended to accommodate a wide variety of mission modules, ranging from route clearance and breaching; logistics; reconnaissance, surveillance, and target acquisition (RSTA); casualty evacuation; and direct- and indirect-fire weapons.
As part of efforts to develop innovative concepts of operation, the Marines are exploring a Fully Autonomous First Wave Concept whereby robotic and autonomous aerial, amphibious, and ground platforms would be employed as the first wave of an amphibious assault to address enemy anti-access/area denial (A2/AD) capabilities. As part of the Marines' Expeditionary Advance Base Operations (EABO), implementation of Fully Autonomous First Wave could result in the need for fewer Marines to participate in high-risk amphibious assault operations, as well as a reduction in the logistical footprint needed to support Marine amphibious operations.
A fully manned, capable and well-trained workforce is a key component of military readiness. The integration of RAS and AI into military units raises a number of personnel-related issues that may be of interest to Congress.
Yet the military's growing body of experience shows that autonomous systems don't actually solve any given problem, but merely change its nature. It's called the autonomy paradox: The very systems designed to reduce the need for human operators require more manpower to support them.
Another consideration revolves around assessments of risks associated with potential failure of RAS and AI systems. National security leaders may want to retain manpower to provide certain capabilities in the event RAS and AI systems are degraded or inadequate for a given mission or requirement.
As the military integrates RAS and AI into its formations, the need to recruit and retain those with advanced technical training will increase. At least some military personnel will need to have a sophisticated understanding of RAS and AI to assist with the design, acquisition, programming, testing, and quality control of such systems. A more sizable population will need to have fairly extensive knowledge of particular systems to effectively use and maintain them. Recruiting and retaining the types of individuals needed to perform these roles may be challenging. They will likely need to have high cognitive ability and years of advanced technical training, in addition to meeting the physical and behavioral standards for joining the armed forces. In addition, the military will likely face intense competition from the private sector for the individuals who do meet all of these requirements.
As RAS and AI become integrated into military formations, the need to train servicemembers on how to use and maintain such systems will increase. While the scope of training required will vary depending on the types of systems introduced and the work roles of individuals, the time and expenses required to develop appropriate training curricula and to train individuals will likely be significant. Such training will not be a one-time event, but will need to be ongoing to accommodate the fielding of new systems and upgrades to older systems. At present, the armed forces provide most military skill training through an extensive network of military schools and training sites. However, this might not be the optimal way to develop and maintain proficiency in RAS and AI systems, particularly for those whose work roles require current, high-level knowledge.
Most individuals who join the armed forces come in at the lowest enlisted or officer grade (rank) and are gradually promoted to higher grades over the course of their military career. These career paths are well defined, and a close connection is typically maintained between one's grade and technical skill, professional military knowledge, leadership experience, authority, status, and compensation. For most military personnel, the career path takes an individual from a focused specialist in a given skill set to a leader of increasingly larger organizations, which themselves are increasingly complex in terms of personnel, systems, and capabilities. Various levels of professional military education, as well as broadening assignments, are designed to facilitate this development.
However, maintaining the desired supply of individuals with expertise in RAS and AI may require a different career path model—for example, higher entry level grades or a technical career track that requires fewer assignments to leadership positions. In addition, in light of rapid technological change, maintaining expertise in RAS and AI might be enhanced by periodic assignments outside of the military—for example, in the private sector or academia. One career path model that has been cited as potentially applicable to those with advanced technical training is the one which exists in the military medical community.106 In this model, officers are routinely brought in at a higher grade based primarily on their medical skills, with less emphasis placed on developing professional military skills and large scale organizational leadership. However, it is not yet clear that this model would be ideal for RAS and AI experts.
The proper test, then, is whether, in some or all circumstances, an autonomous weapons system is indiscriminate by nature or is of such a nature as to cause superfluous injury or unnecessary suffering, or is likely to cause widespread long-term and severe damage to the environment. In other words, an autonomous weapons system, like all other weapon systems, must be able to distinguish targets appropriately and to ensure that the attack is proportionate to the expected military advantage. Some argue that the fact that the weapon system operates autonomously does not per se govern whether it can meet these criteria.139 As the argument goes, as long as the autonomous system can be programmed with sufficiently reliable targeting information to ensure that it can be fired at a lawful target, the autonomous weapon system can survive an Article 36 review.140 On the other hand, some scientists have warned that robotic systems capable of distinguishing between civilians and combatants or making proportionality judgments are not likely in the short term.141 The ban on weapons that cause superfluous or unnecessary damage or suffering likewise arguably can be met by, among other things, ensuring the payload is not a prohibited type, such as biological or chemical in nature.
(a) Function as anticipated in realistic operational environments against adaptive adversaries.
(b) Complete engagements in a timeframe consistent with commander and operator intentions and, if unable to do so, terminate engagements or seek additional human operator input before continuing the engagement.
(a) Be readily understandable to trained operators.
(b) Provide traceable feedback on system status.
Notable futurists, such as Elon Musk and the late Steven Hawking, have warned that AI has the potential to be far more dangerous than nuclear weapons158 and could be "the worst event in the history of our civilization."159 While of note perhaps more important is a growing level of organized groups concerned with LAWS. The following are examples of organized groups concerned with lethal autonomous weapon systems (LAWS).
In April 2017, DOD established the Algorithmic Warfare Cross-Functional Team to oversee Project Maven, a project to use AI to autonomously extract objects of interest from moving or still UAV imagery. Project Maven would develop computer vision algorithms, trained using AI techniques such as machine learning, to better identify targets. Tools built through Project Maven would be used to complement the labor-intensive process of humans analyzing drone video, a practice that delays providing results to warfighters for targeting purposes. It was suggested that such AI-enhanced tools could allow human analysts to process up to two to three times as much data within the same time period, providing more time-sensitive targeting data and, according to DOD, a reduction of collateral damage and civilian casualties.
Google is one of a number of companies involved in Project Maven. In March 2018, Google's role in Project Maven came to light and a number of Google employees reportedly expressed concern "that the company would offer resources to the military for surveillance technology involved in drone operations" while "others argued that the project raised important ethical questions about the development and use of machine learning."161 Reportedly about a dozen Google employees resigned in protest and about 4,000 employees signed a petition demanding "a clear policy stating that neither Google nor its contractors will ever build warfare technology."162 Reportedly Google will not renew its contract with DOD for Project Maven when the current contract expires in 2019.
On September 12, 2018, Members of the European Parliament overwhelming passed a nonbinding resolution urging Member States and the European Council to work toward an international ban on weapon systems that lack meaningful human control over the critical function of selecting and engaging targets. It was also noted that France and Germany are not working toward and do not support the strict and total prohibition of autonomous weapons.
A lack of trust by the Warfighters, and the wider public, is a major roadblock in DOD's continued development and use of autonomous unmanned systems. This lack of trust is highlighted by international efforts at the United Nations (U.N.) to consider policies that would prohibit the deployment of autonomous systems with lethal capabilities. Additionally, there are technological shortcomings in the current abilities of AI regarding ethical thinking that may limit the public's trust of autonomous unmanned systems developed for military capabilities.
In considering the specific use of weaponized systems, Department of Defense Directive (DoDD) 3000.09, Autonomy in Weapon Systems, signed in November 2012, established policies and assigned responsibilities to shape the development and use of autonomous functions in weapon systems, including manned and unmanned platforms. It mandates that autonomous and semi-autonomous weapon systems be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force. DoDD 3000.09 also requires that persons who authorize the use of, direct the use of, or operate autonomous and semi-autonomous weapon systems must do so with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable ROE. DoDD 3000.09 underwent a mandatory periodical update with administrative changes (Change 1, May 8, 2017), but a more substantive update was expected to be completed in late 2017. That substantive update, when released, is expected to involve clarifications of definitions and processes rather than a shift in the overall thrust of the policy.
DOD does not currently have an autonomous weapon system that can search for, identify, track, select, and engage targets independent of a human operator's input. These tasks currently rely heavily on a human operator using remote operation, also referred to as "tele-operation." In the future, weaponization will be a crucial capability in mission sets where the unmanned system is directly supporting forces engaging in hazardous tasks.
In view of this policy position and, in consideration to the challenges posed by a possible treaty regulating AI, this suggests future DOD development of LAWS is possible, given favorable advances in AI technology and an absence of international restrictions.
It has been suggested that military RAS and AI have the potential to change the very nature of war.176 At the strategic level, this could affect how the U.S. organizes ground forces, how it fights, and what types of major weapon systems it will need. At the operational and tactical levels, enemy RAS and AI capabilities could dictate specific weapon systems design, the development of new types of units to address enemy RAS and AI, and how brigade to squad level units conduct tactical operations.
While the DOD and the Services emphasizes the development of RAS and AI for the U.S. military, it might be considered prudent to have a full understanding of foreign military RAS and AI efforts to help guide U.S. efforts and possibly preclude strategic or tactical RAS and AI "surprises" posed by foreign militaries or nonstate groups.
While DOD continues to eschew fully automated ground combat systems, there is a possibility that other nations could aggressively pursue fully autonomous weapons systems to achieve a battlefield advantage. This raises a question: are other nation's efforts to develop LAWS sufficient justification for the United States to do the same? The evolution of other weapons, such as hypersonic weapons, could also result in U.S. development of fully autonomous weapon systems from a purely defensive perspective. Despite DOD's insistence that a "man in the loop" capability will always be part of RAS systems, it is possible if not likely, that the U.S. military could feel compelled to develop ground fully autonomous weapon systems in response to comparable enemy ground systems or other advanced threat systems that make any sort of "man in the loop" role impractical.
While U.S. development of RAS and AI for use by ground forces is articulated in a variety of strategies and directives, it can be argued that equally and, possibly, more important is the development of technologies, systems, formations, and tactics, techniques, and procedures designed to counter foreign use of RAS and AI against U.S. ground forces. Such a "defensive" approach to RAS and AI could prove to be particularly prudent, as it is possible other nations may introduce RAS and AI systems into ground combat before the United States, thereby necessitating an effective means of countering their use. Furthermore, foreign RAS and AI advances not directly related to weapons such as AI-enhanced military decisionmaking and intelligence or RAS-enabled logistics could have an indirect impact on U.S. ground forces and may be taken into consideration when developing U.S. counter RAS and AI capabilities.
Many cutting-edge innovations in AI and RAS are occurring in the private sector, and these innovations could be adapted for use by U.S. ground forces. However, engaging with the private sector raises policy considerations for the Army and Marine Corps, as well as DOD more broadly. For example, if policymakers seek to assist the military in making use of emerging technologies such as AI and RAS, they might consider expanding support for entities such as the Defense Innovative Unit. If the Administration or Congress seeks to have more in-house expertise in AI and RAS, they might also consider supporting the development of service incentives or recruitment models to attract additional experts, either on a permanent or temporary basis.
Where will RAS and AI be integrated into the military most rapidly?
Which occupational specialties will be less needed due to replacement by RAS and AI?
Which occupational specialties will be in greater demand as RAS and AI are integrated into the force?
What degree of "human backup" is desired in the event RAS and AI systems do not perform as required?
Which particular skills do the military services require to support the integration of RAS and AI systems?
What fields of academic study and work history are most essential to target for recruiting and retention?
The armed forces have any array of special and incentive pay authorities that allow them to increase compensation for targeted skill sets and assignments. Are these authorities sufficient? Are the Services using them effectively?
What nonmonetary incentives are most attractive to individuals with those skills? Do the armed forces have the authority to offer those incentives?
How valuable are those skills to private-sector employers? How competitive are the armed forces in competing for this type of labor?
Should certain eligibility criteria for joining the armed forces be modified to make military service a more viable option for those with the necessary skills?
To what extent can other federal civilians and contractors with advanced technical skills be substituted for military personnel with those skills?
How much training will the armed forces have to provide to their work force can effectively employ RAS and AI systems?
When and where will this training occur, and at what cost?
How quickly will the armed forces be able to adapt their training programs to account for technical advances in RAS and AI, and new concepts of operations?
What role might private-sector training or training with industry play, particularly for those whose work roles require the most sophisticated knowledge of RAS and AI?
What assignments will be most critical to the professional development of RAS and AI experts and their ability to contribute to operational missions? Might private-sector or academic assignments be part of the career path?
Will specialists in RAS and AI be expected to lead units and, if so, of what size and complexity? Does a technical track make sense?
How much professional military knowledge will RAS and AI specialists need to possess to effectively integrate into military organizations? How much knowledge of warfighting functions will they need to advise commanders on how best to employ RAS and AI and how to synchronize them with other systems for maximum effect?
In view of this, there appears to be an opportunity for Congress to examine these issues and perhaps define its role in the overall U.S. governmental debate on LAWS. These debates and resulting policy decisions and guidance could prove beneficial to DOD as a whole—and the Army and Marines in particular—as they develop RAS and AI systems and capabilities for use on the battlefield.
Such restrictions could limit other nations as well. Should deliberate formal efforts to regulate LAWS emerge in the coming years, the United States has the option of playing an active role, a passive role, or no formal role whatsoever. Because of the potential national security implications of a potential LAWS ban, as well as its impact on Army and Marine RAS and AI developmental efforts, policymakers both in the Administration and Congress might examine and decide what the appropriate level of U.S. involvement should be in such an international regulatory process.
Adam Wunische, "AI Weapons Are Here to Stay," The National Interest," August 7, 2018.
Summary of the 2018 National Defense Strategy of the United States of America, Sharpening the American Military's Competitive Edge, 2018, p. 5.
Washington, DC, May 18, 2009, p. 41.
Taken directly from Joint Chiefs of Staff, Joint Concept for Robotic and Autonomous Systems (JCRAS), October 16, 2016, p. 2.
Taken directly from M.L. Cummings, Artificial Intelligence and the Future of Warfare, Chatham House, The Royal Institute of International Affairs, January 2017, p. 2.
See the FUTURE of Artificial Intelligence Act (S. 2217 and H.R. 4625), the AI JOBS Act of 2018 (H.R. 4829), and the John S. McCain National Defense Authorization Act for Fiscal Year 2019 (P.L. 115-232). The AI classification scheme is presented in Peter Norvig and Stuart J. Russell, Artificial Intelligence: A Modern Approach, 3rd ed. (New Jersey: Prentice Hall, 2009).
For an overview of AI and selected, overarching issues and policy considerations, see CRS In Focus IF10608, Overview of Artificial Intelligence, by Laurie A. Harris.
Taken directly from Expert System, https://www.expertsystem.com/machine-learning-definition/, accessed August 30, 2018.
Taken directly from Department of Defense (DOD) Directive 3000.09, Change 1, May 8, 2017, Autonomy in Weapons Systems, pp. 13-14.
For additional information, see CRS Recorded Event WRE00207, Artificial Intelligence: Innovation, Impacts, & Policy Considerations for the 115th Congress, by Laurie A. Harris.
Wisskirchen et al., "Artificial Intelligence and Robotics and Their Impact on the Workplace," International Bar Association Global Employment Institute, April 2017, p. 9.
The report's authors consider the First Industrial Revolution as the beginning of the Industrial Age around 1800, the Second Industrial Revolution as the beginning of electrification at the end of the 19th century, and the Third Industrial Revolution as the beginning of digitization in the 1970s.
For a detailed discussion of AI and national security, see CRS Report R45178, Artificial Intelligence and National Security, by Daniel S. Hoadley and Nathan J. Lucas.
Summary of the 2018 National Defense Strategy of the United States of America: Sharpening the American Military's Competitive Edge, p. 7.
The Army Vision (2018-2028), https://www.army.mil/article/206270/the_army_vision_2018_2028, June 6, 2018.
Paul Scharre, Army of None: Autonomous Weapons and the Future of War, 2018, p. 102.
Sydney J. Freedberg Jr., "Evolutionary Revolution: Esper's Army Modernization Philosophy," BreakingDefense.com, May 16, 2018.
Mick Ryan, Human-Machine Teaming for Future Ground Forces, Center for Strategic and Budgetary Assessments, 2018, p. 9.
For a partial listing of these studies, see, for example, Erin Winik, "Every study we could find on what automation will do to jobs, in one chart," MIT Technology Review, January 25, 2018.
Pew Research Center, AI, Robotics and the Future of Jobs, August 2014, p. 5.
David H. Autor, "Why Are There Still So Many Jobs? The History and Future of Workplace Automation," Journal of Economic Perspectives, Vol. 29, No. 3, Summer 2015, pp. 3-30.
Jaime Condliffe, "Could the AI Talent Shortage Be Eased If Tech Giants Learn to Share?" MIT Technology Review, November 10, 2017, and Jacques Bughin, Eric Hazan, Sree Ramaswamy, et al., How Artificial Intelligence Can Deliver Real Value to Companies, McKinsey Global Institute, June 2017.
Isaac R. Porche III, Bradley Wilson, Erin-Elizabeth Johnson, "How do We Deal with a Flood of Data," RAND, June 23, 2014.
Richard Danzig, "Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority," Center for New American Security, June 2018, p. 15.
Information in this section is taken from Mick Ryan, Human-Machine Teaming for Future Ground Forces, Center for Strategic and Budgetary Assessments, 2018, pp. 11-14.
Dr. Bob Sadowski, "Shaping the Future: Army Robotics and Autonomous Systems," U.S. Army Tank Automotive Research, Development and Engineering Center, March 2016, p. 6.
Jason Sherman, "Army Previews Desired Capabilities for Next Generation Combat Vehicle," Inside the Army, August 31, 2018.
Victoria Leoni, "Here's How Artificial Intelligence Could Predict When Your Army Vehicle Will Break Down," Military Times, June 27, 2018.
RAND's 2013 report: The Army's Role in Overcoming Anti-Access and Area Denial Challenges," p. ix characterizes A2/AD as "Anti-access (A2) challenges prevent or degrade the ability to enter an operational area. These challenges can be geographic, military, or diplomatic. Area denial (AD) refers to threats to forces within the operational area. As they relate to U.S. ground forces (the Army and Marine Corps), AD threats are characterized by the opponent's ability to obstruct the actions of U.S. forces once they have deployed."
Information provided to CRS from Army Staff, October 5, 2018.
Paul Mozor, "Beijing Wants A.I. to Be Made in China by 2030," New York Times, July 20, 2017.
"The Race for AI," Defense One, March 2018, Forward.
Patrick Turner, "A Criminal Gang Used a Drone Swarm to Obstruct an FBI Hostage Raid," Defense One, May 3, 2018.
Nikolai Novichkov, "New Russian Combat UGV Breaks Cover, Uran-9 Readies for Service," IHS Jane's International Defence Review, October 2016, p. 30.
Kelsey Atherton, "Russia Confirms its Armed Robot Tank was in Syria," C4ISRNet.com, May 7, 2018.
Samuel Bendett, "Here's How the Russian Military is Organizing to Develop AI," Defense One, July 20, 2018.
"The Race for AI," Defense One, March 2018, p. 14.
Gabriel Dominguez, "China Seeking to Surpass U.S. in Military Technologies, Says Canadian Report," Jane's Defence Weekly, June 6, 2018, p. 6.
Gabriel Dominguez, "China Threatening U.S. DOD's Technical, Industrial Base, Say Pentagon Officials," Jane's Defence Weekly, June 27, 2018, p. 8.
CRS meeting with Army Staff, July 24, 2018.
Tim Hornyak, "Korean Machine Gun Robots Start DMZ Duty," C/NET.com, July 14, 2010.
"Britain Uses Brimstone Missiles for the First Time in Syria," Indian Defense News, January 12, 2016.
Office of the Secretary of Defense, Unmanned Systems Integrated Roadmap 2017-2042, June 2018, p. 1.
Joint Concept for Robotic and Autonomous Systems (JCRAS), October 16, 2016, p. v.
Department of Defense (DOD) Directive 3000.09, Change 1, May 8, 2017, Autonomy in Weapons Systems, p. 1.
Senate Committee on Armed Services, Hearing to Consider the Nomination of General Paul J. Selva, USAF, for Reappointment to the Grade of General and Reappointment to be Vice Chairman of the Joint Chiefs of Staff, 115th Congress, 1st Session, July 18, 2017.
2018 Department of Defense, Artificial Intelligence Strategy, June 2018.
Deputy Secretary of Defense Memorandum, Subject: Establishment of the Joint Artificial Intelligence Center, June 27, 2018.
Justin Doubleday, "Military's New AI Center Aims to Accelerate Fielding, Attract "World-Class" Talent," InsideDefense.com, July 12, 2018.
For additional information on Army Futures Command, see CRS Insight IN10889, Army Futures Command (AFC), by Andrew Feickert.
Army Directive 2018-18, Army Artificial Intelligence Task Force in Support of the Department of Joint Artificial Intelligence Center, October 2, 2018.
Information in this section is from a CRS meeting with Marine Corps Staff, September 4, 2018.
U.S. Army Training and Doctrine Command, The U.S. Army Robotic and Autonomous Systems Strategy, March 2017, http://www.arcic.army.mil/App_Documents/RAS_Strategy.pdf.
Marine Corps Robotic and Autonomy Strategy (MCRAS), June 2018.
M.L. Cummings, Artificial Intelligence and the Future of Warfare, Chatham House, The Royal Institute of International Affairs, January 2017, pp. 4 and 9-10.
Information in this section is taken from U.S. Army Briefing, Army Ground Robotics Update, November 14, 2017, and Melanie Rovery, "SMET Advances: U.S. Army Selects Its Robot Mules for the Next Phase," Jane's International Defence Review, March 2018.
Information in this section was provided to CRS by the Army Staff, October 5, 2018.
Army Briefing, PM Force Projection Overview, February 28, 2017.
Information in this section is from Vivienne Machi, "Army Rolling Ahead with Manned-Unmanned Convoys," National Defense, April 4, 2018.
Army Briefing, Army Robotic and Autonomous Systems (RAS) Portfolio, April 11, 2018.
Army Briefing, Army Ground Robotics Overview: OSD Joint Technology Exchange Group, April 24, 2018.
Sydney J. Freedberg Jr., "Army Pushes Bradley Replacement; Cautious on Armed Robots," Breaking Defense, June 27, 2018.
Sean Kimmons, "Initial Prototypes for Next-Generation Combat Vehicle to Focus on Manned-Unmanned Teaming," Army News Service, March 26, 2018.
Ibid. and Army Briefing, Army Robotic and Autonomous Systems (RAS) Portfolio, April 11, 2018.
Ashley Tressel, "DIUx Awards Contract to Use AI Software for Bradley," Insidedefense.com, June 26, 2018.
James Vincent, "The U.S. Army is Using Machine Learning to Predict When Combat Vehicles Need Repair," The Verge, June 26, 2018.
Jack Corrigan, "How Artificial Intelligence Could Keep U.S. Army Vehicles Ready for Action," Defense News, August 11, 2018.
Aaron Gregg, "Army to Use Artificial Intelligence to Predict Which Vehicles Will Break Down," The Washington Post, June 26, 2018.
"The Autonomy Paradox: Why 'Unmanned Systems' Don't Shrink Manpower Needs," Armed Forces Journal, (October, 2011), http://armedforcesjournal.com/the-autonomy-paradox/.
The largest exception to this is the military medical community, many of who join the armed forces after completing extensive civilian schooling and training. Lawyers and chaplains are also typically recruited after completion of additional schooling or training.
Secretary of Defense Memorandum, The Next Two Links to the Force of the Future, June 9, 2019.
See International Committee of the Red Cross, Autonomous weapon systems: Technical, military, legal and humanitarian aspects, Expert Meeting, Geneva, Switzerland, 26-28 March 2014, at 61 ("In theory, it would only be one step further [from current unmanned but remotely controlled weapons systems] to allow a mobile unmanned system to fire a weapon without human intervention, but one that would entail a technological leap in capability while simultaneously raising significant legal and ethical questions."). With respect to unmanned aerial vehicles, DOD has predicted that "the level of autonomy should continue to progress from today's fairly high level of human control/intervention to a high level of autonomous tactical behavior that enables more timely and informed human oversight." Markus Wagner, The Dehumanization of International Humanitarian Law: Legal, Ethical, and Political Implications of Autonomous Weapon Systems, 47 Vand. J. Transnat'l L. 1371, 1374 (2014) (quoting FY 2009-2034 Unmanned Systems Integrated Roadmap 27 (2009)).
See. e.g., Kenneth Anderson and Matthew Waxman, Law and Ethics for Autonomous Weapon Systems Why a Ban Won't Work and How the Laws of War Can 4 (Hoover Institution and Stanford University 2013) (recommending high-level weapons reviews to determine propriety of autonomous weapons systems on a case-by-case basis).
See Christof Heyns, Autonomous Weapons Systems: Living a Dignified Life and Dying a Dignified Death, in Autonomous Weapons Systems 3, 10 (Nehal Bhuta, et al., eds. 2016) (arguing that delegating to a machine the decision to use deadly force against a human being could be an inherently arbitrary deprivation of the right to life). For an alternate view, see Dieter Birnbacher, Are Autonomous Weapons a Threat to Human Dignity, in Autonomous Weapons Systems 105, 120 (Nehal Bhuta, et al., eds. 2016) (arguing that "for victims whose dignity is at stake, it is a matter of indifference whether the threat they are exposed to comes from manned or unmanned weapons…").
See Mary Ellen O'Connell, Banning Autonomous Killing: The Legal and Ethical Requirement that Humans Make Near-Time Lethal Decisions, in American Way of Bombing: Changing Ethical and Legal Norms, from Flying fortresses to Drones 224, 236 (Matthew Evangelista and Henry Shue eds., 2014) (arguing that it is imperative that human beings retain sovereignty over kill decisions and advocating a treaty banning fully autonomous killing).
See Heyns, supra note 109, at 13 (describing emerging view that the central question as to the legality of autonomous weapons systems will turn on whether such systems allow "meaningful human control" over the decision to use force against a human being).
See Guglielmo Tambunrrini, On Banning Autonomous Weapons Systems from Deontological to Wide Consequentialist Reasons, in Autonomous Weapons Systems, supra note 109, at 122, 131 (arguing that current state-of-the-art and foreseeable developments in robotics and AI "overwhelmingly support the moral obligation of suspending autonomous weapons systems development" until compliance with LOAC principles can be convincingly demonstrated).
See Anderson and Waxman, supra note 108, at 21 (arguing there may be "serious humanitarian risks to prohibition [of autonomous weapons systems], given the possibility that autonomous weapons systems could in the long run be more discriminating and ethically preferable to alternatives").
Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflict, adopted June 8, 1977, 1125 U.N.T.S. 3, entered into force December 7, 1978, art. 36 (hereinafter API). The United States is not a party to API but may regard parts of it as representing customary international law. See Anderson and Waxman, supra note 104, at 9; Wagner, supra note 103, at 1385 (noting that the United States has incorporated many provisions of API into its military manuals).
Jean-Marie Henckaerts and Louise Doswald-Beck, I Customary International Humanitarian Law 3 (2005) (hereinafter "ICRC Study") Rule 1 states that "The parties to the conflict must at all times distinguish between civilians and combatants…." Rule 7 states that "The parties to the conflict must at all times distinguish between civil objects and military objectives…." Id. at 25.
Id. at 46 (Rule 14 states "Launching an attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated is prohibited).
Wagner, supra note 107, at 1388 (positing that computers, which may be capable of computing quantitative assessments, are currently less capable of making qualitative assessments, and questioning whether "the principle of distinction and the principle of proportionality can be encoded into digital format").
Id. at 1403 (noting that culpability requires moral agency, which machines can never possess, and that a traditional deterrence rationale does not apply).
API, supra note 115, art. 52(2).
Wagner, supra note 107, at 1390.
API, supra note 115, art. 52(3).
Wagner, supra note 107, at 1396.
API, supra note 115, art. 57 (requiring, among other things, that those responsible for military operations "refrain from deciding to launch any attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated").
Regulations concerning the Laws and Customs on Land, Annex to Convention IV Respecting the Laws and Customs of War on Land art. 22 (1907) (hereinafter Hague Regulation); API, supra note 115, art. 35(1).
API, supra note 115, art. 35(2)-(3).
See, e.g., Convention on the Prohibition of the Development, Production, Stockpiling, and Use of Chemical Weapons and on their Destruction, Apr. 29, 1997, 1974 U.N.T.S. 45 (hereinafter CWC); Convention on the Prohibition of the Development, Production and Stockpiling of bacteriological (Biological) and Toxin Weapons and on their Destruction, Mar. 26, 1975, 1975 UST 583, 1015 UNTS 163 (hereinafter BWC). For a list of other types of weapons that may be banned under customary international law, see ICRC Study, supra note 116, part IV.
ICRC Study, supra note 116, at 40 (Rule 12 describes indiscriminate attacks as those "(a) which are not directed at a specific military objective; (b) which employ a method or means of combat which cannot be directed at a specific military objective; or (c) which employ a method or means of combat the effects of which cannot be limited as required by international humanitarian law; and consequently, in each such case, are of such a nature to strike military objectives and civilians or civilian objects without distinction"). See also id. at 244 (Rule 71 states that "the use of weapons which are by nature indiscriminate is prohibited").
API, supra note 115, art. 36.
Anderson and Waxman, supra note 108, at 10.
See Tambunrrini, supra note 112, at 130 (citing profound technological challenges to be resolved, such as effective in situ target discrimination and recognition of the status of those otherwise hors de combat") citing Ronald Arkin, Governing Lethal Behavior in Autonomous Robots (2009)).
Department of Defense (DOD) Directive 3000.09, Change 1, May 8, 2017, Autonomy in Weapons Systems.
Dan Saxon, A Human Touch: Autonomous Weapons, DoD Directive 3000.09 and the Interpretation of 'Appropriate Levels of Human Judgment over the Use of Force', 165, 165-66 in Autonomous Weapons Systems (Nehal Bhuta, et al., eds. 2016).
Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, A/HRC/23/47, at 11 (2013) (Th[e] ease [of deploying lethal autonomous robotic systems [LARS] rather than human soldiers] could potentially affect political decisions. Due to the low or lowered human costs of armed conflict to States with LARs in their arsenals, the national public may over time become increasingly disengaged and leave the decision to use force as a largely financial or diplomatic question for the State, leading to the "normalization" of armed conflict.").
Anderson and Waxman, supra note 108, at 18 (arguing that "this concern is not special to autonomous weapons. The same objection has already been made with respect to remotely-piloted UAVs and high-altitude bombing before that. Generally, it can be made with respect to any technological development that either reduces risk to one's own forces or reduces risk to civilians, or both").
Id. at 7 (noting that "no one seriously expects remotely-controlled or autonomous systems to completely replace humans on the battlefield. Many military missions will always require humans on the ground, even if in some contexts they will operate alongside and in conjunction with increasingly automated, sometimes autonomous, systems").
HRW, supra note 147, at 38 (Robots cannot identify with humans, which means that they are unable to show compassion, a powerful check on the willingness to kill. For example, a robot in a combat zone might shoot a child pointing a gun at it, which might be a lawful response but not necessarily the most ethical one.").
See Tambunrrini, supra note 112, at 129 (noting technical challenge associated with programming a robot to recognize persons who are hors de combat, which may require "viewpoint independent classification of bodily postures and gestures in variable illumination conditions, in addition to an understanding of emotional expressions and real-time reasoning about deceptive intentions….").
Anderson and Waxman, supra note 108, at 15 ("We should not rule out in advance possibilities of positive technological outcomes—including the development of technologies of war that might reduce risks to civilians by making targeting more precise and firing decisions more controlled (especially compared to human-soldier failings that are so often exacerbated by fear, panic, vengeance, or other emotions—not to mention the limits of human senses and cognition).").
Noel Sharkey, Staying in the Loop: Human Supervisory Control of Weapons, in Autonomous Weapons Systems 23, 35 (Nehal Bhuta, et al., eds. 2016) ("The operation of automatic reasoning has been shown to favor the uncritical acceptance of suggestions and maintains a strong bias.").
P.W. Singer, Wired for War 395 (2005) (arguing that "technology might lessen the likelihood of anger-fueled rage, but also make some soldiers too calm, too unaffected by killing").
Catherine Clifford, "Elon Musk: Mark my Words—AI is Far More Dangerous than Nukes," CNBC, March 13, 2018.
Arjun Kharpal, "Steven Hawking Says AI Could be Worst Event in the History of Our Situation," CNBC, November 6, 2017.
Information in this section is taken from Cheryl Pellerin, "Project Maven to Deploy Computer Algorithms to War Zone by Year's End," DOD News, June 21, 2017; Kelsey Atherton, "Targeting the Future of DOD's Controversial Project Maven Initiative, C4ISRnet, July 27, 2018; Nick Statt, "Google Reportedly Leaving Project Maven Military AI Program After 2019, The Verge, June 1, 2018; and Daisuke Wakabayashi and Scott Shane, "Google Will Not Renew Pentagon Contract That Upset Employees," New York Times, June 1, 2018.
https://www.stopkillerrobots.org/, accessed August 31, 2018.
Michael C. Horowitz and Julia Macdonald, "Will Killer Robots be Banned? Lessons from Past Civil Society Campaigns," Lawfare, November 5, 2017.
Sean Welsh, "Kill Switch," Jane's Intelligence Review, March 2018, p. 51.
European Parliament, 2014-2019, Motion for a Resolution on Autonomous Weapon Systems, September 5, 2018 and Brooks Tigner, "MEPs Lobby for Total International Ban on Lethal Autonomous Weapons," Jane's Defence Weekly, September 19, 2018, p. 4.
Office of the Secretary of Defense, Unmanned Systems Integrated Roadmap 2017-2042, June 2018, pp. 21-22.
William A. Galston, "Why the Government Must Help Shape the Future of AI," Brookings Institute, September 13, 2018.
Subcommittee on Information Technology, Committee on Oversight and Government Reform, U.S. House of Representatives, "Rise of the Machines: Artificial Intelligence and its Growing Impact on U.S. Policy," September 2018, p. 6.
The Race for AI, Defense One, March 2018, p. 10.
See CRS Report R45178, Artificial Intelligence and National Security, by Daniel S. Hoadley and Nathan J. Lucas and Peter Singer; "Getting to Grips with Military Robots," The Economist, January 25, 2018; and Aaron Mehta, "AI Makes Mattis Question Fundamental Beliefs About War," C4ISRnet.com, February 17, 2018.
David Barno and Nora Bensahel, "War in the Fourth Industrial Revolution," War on the Rocks, June 19, 2018.
Jade Leung and Sophie-Charlotte Fischer, "JAIC: Pentagon Debuts Artificial Intelligence Hub," Bulletin of the Atomic Scientists," August 8, 2018, at https://thebulletin.org/2018/08/jaic-pentagon-debuts-artificial-intelligence-hub/.
Office of the Secretary of Defense, Unmanned Systems Integrated Roadmap 2017-2042, June 2018, pp. 22-23.

References: v.

 art. 36
 art. 52
 art. 52
 art. 57
 art. 22
 art. 35
 art. 35
 art. 36