Certification of Autonomous Systems under Uk Military Safety Standards

There is growing interest in many sectors in developing highly autonomous systems such as Unmanned Air Vehicles (UAVs) with significant in-mission executive power. Such systems have the potential to replace humans in a variety of dangerous tasks, but there is concern that the combination of novel technologies with demanding tasks and unpredictable environments will lead to new safety challenges. This paper reviews proposed scenarios of autonomous system applications and identifies the safety concerns that they raise. It then explores how autonomous systems can be certified as safe to operate within the terms of the existing safety standards that are used by the UK military. The combination of difficulties arising from the nature of autonomous systems and the provisions of the existing safety standards raise serious concerns about the practicality of certification. With this in mind, promising work on new techniques for analysing and ensuring the safety of autonomous systems is reviewed. Finally, some future concerns and directions are identified. Introduction The SEAS DTCT is a project funded by the UK Ministry of Defence with the aim of with developing technologies and methods for building autonomous systems (AS) which will operate in a range of military roles. Such systems clearly have the potential for life-threatening accidents. As part of the SEAS effort, the authors have reviewed the current situation on safety certification of AS. A system capable of causing an accident that leads to human injury or death, or substantial material loss, is considered safety-critical, and before being deployed it must be certified as adequately safe according to applicable standards. The standards that apply vary with the type of system and the environment in which it will be operated. For example, the safety-critical systems procured and operated by the UK Ministry of Defence must now be certified against the requirements given by Def Stan 00-56 [1]. The need to certify autonomous systems is new, and consequently there is neither an established way of performing certification nor adequate advice on how this should be achieved. The specific technologies used in AS, and the complex environments in which they must perform, present further difficulties. The next section uses the set of vignettes that have been defined for the SEAS DTC to sketch some ways in which AS can be dangerous. This is followed by an exploration of why AS present problems for safety engineering. Relevant safety standards are then reviewed. The safety problems and certification requirements are drawn together to present some requirements for moving forward, and finally a selection of existing work is reviewed in the light of these. Risks in the DTC Vignettes There have been a number of vignettes developed for use by the SEAS DTC projects. There are four main accident types that can occur in the vignettes: 1. Collision of autonomous vehicle (AV) with human pedestrian or vehicle with human occupant (or near miss, causing said vehicle to crash). 2. Human hit by AS combat capability. 3. Human exposed to threat due to AS inadequately or inaccurately reporting a threat. 4. AS action causes/triggers accident outside of its own capability. 1 Systems Engineering for Autonomous Systems Defence Technology Centre Vign. No. Vignette Objective AVs Hazard Accident Type 7 Harbour Reconnaissance Locate and recover cargo containers containing hazardous material in a flooded area. USVs, UUVs AV fails to recognise leaking container; exposes survivors to contamination. 4 8 Air Attack Locate and neutralise an unknown number of SAM sites in enemy territory before troops are called in. UAVs Misidentification of a civilian site as a SAM installation. 2 9 Urban Reconnaissance Ensure house is clear (e.g. free of insurgents, booby traps and hazardous chemicals) and safe for troops to enter. UGVs Incorrect mapping could mean that a room containing a threat is missed, endangering troops. 3 10 Route Maintenance Patrol route, reporting threats such as mines, car bombs and snipers to command. UGVs AV is travelling on the wrong side of the road, whilst a humanoccupied vehicle is approaching. 1 Table 1 – Example Hazards for Vignettes 7-10 While the first two accident types on the previous page are direct consequences of the failure of a safety-critical function, type 3 can be caused by a failure to perform a safety-critical function at the system of systems level – this may be an example of poor reliability or availability leading to a safety problem. Meanwhile, type 4 is indicative of a hazard present in the environment, as opposed to one that an AS can cause on its own. For example, an AV runs over and inadvertently triggers a mine, or an AS gives a friendly position away to the enemy. Table 1 shows some examples of the above-mentioned hazards in vignettes 7 through 10.