SL Towards a Trustworthy Foundation for Assured Autonomous Systems Mr

Many modern information systems mitigate cyber security risks by making use of commercial software security technologies that continually monitor behaviour and apply updates. Unmanned and/or Autonomous (UA) system implementations prove substantial challenges for both of these mitigation strategies, as network connectivity is limited and unreliable, and system complexity renders patching impractical outside of a routine update process. The task of performing security assessments on UA system implementations is a challenging proposition as UA systems often lack the interfaces by which traditional software testing is applied (e.g., a network server or a humanmachine interface.) Further, many control decisions must be made automatically in the UA system instead of being made by a human. In this paper we provide an in-depth characterization of an alternative, proven approach to UA system design and implementation that reduces risks and costs associated with these challenges. These benefits are realized by making use of technologies and techniques that promote desirable properties such as correct-byconstruction, least privilege, and least functionality while doing so in a manner that more easily facilitates security assessments. 1.0 INTRODUCTION AND CONTEXT Unmanned and/or Autonomous (UA) systems frequently operate in environments where defensive countermeasures are scarce, both with respect to kinetic threats and with respect to cyber security threats. In order to ensure that a UA system can successfully execute its mission, developers must provision these systems with adequately robust cyber defenses such that they can prevent adversaries from compromising the execution control flow of programs within the platform. Unfortunately, cyber defense tactics that are common in typical information systems are difficult to implement in UA systems. Due to the limited bandwidth available during a mission, continuous patching and updates are often impractical, and intrusion detection systems are difficult to monitor in real time. Because the software systems used within a given UA system are typically highly customized, they are frequently not supported by traditional security scanning tools (e.g., NESSUS, STIG) (1). Further, UA systems tend to require higher levels of determinism, and are also frequently constrained by CPU and memory limitations. Even if one overlooks the limited benefits associated with near real-time security scanning (e.g., endpoint protection systems), limited system resources (e.g. memory, CPU) render implementing such security controls implausible. Lacking access to the software security solutions that are used in enterprise IT systems, UA system developers must consider constructing a secure, modular, and testable runtime environment in order to achieve robust cyber security within a UA system. We propose the use of low-level technologies such as bare metal hypervisors, separation kernels, unikernels, real time operating systems (RTOS), and carefully constructed Linux Towards a Trustworthy Foundation for Assured Autonomous Systems 4 2 STO-MP-IST-166 implementations to run software according to the needs of each particular application. These runtimes can then be provisioned to make use of message passing via shared memory regions in order to facilitate partitioned information flow control amongst different software functions that are governed by different safety, security, and reliability policies. In order for a secure runtime to be practical, it must facilitate the implementation of software that effectively implements the functionality desired by the system developers. Similarly, a secure runtime must allow developers to leverage existing Commercial Off The Shelf (COTS), Government Off The Shelf (GOTS), and open source software components when developing and integrating solutions, as frequently existing software has been rigorously assessed and ultimately approved to serve a specific purpose within a system. We propose a development model that makes use of message passing as the sole means for software components to invoke behaviour in other software components (e.g., libraries, processes.) This approach enables integrators to compose systems using heterogeneous technologies and development approaches, while simultaneously enabling developers to use tools and languages that are most suitable to their needs. The decision control system that directs the actions of the UA system present new security concerns. This system must be pre-programmed in the UA system before deployment to recognize the environment from sensory inputs. For example, a UAV may have multiple sensors (e.g., infrared, heat) that it uses to recognize troops and ground vehicles to provide useful reconnaissance. In this case, both the sensors and the machine learning recognition software that processes the inputs both provide adversaries with new attack vectors. Finally, we consider the problem of system assurance. For any given system, claims of robust security must be backed up by measurable evidence in order for stakeholders to have confidence in the presence of that security. In order to generate such evidence of security, a system must be subjected to comprehensive testing and analysis, which typically must include independent 3rd party testing and analysis. Considerations for how UA systems can be designed and provisioned for these assessments is briefly described. 2.0 PROBLEM STATEMENT Unmanned and/or Autonomous (UA) Systems perform increasingly critical roles within today’s military operations. As operations become dependent on UA systems for success, it is essential that self-protection countermeasures implemented in these systems evolve at a more rapid pace than countermeasures that are meant to degrade UA system effectiveness. With respect to cyber security, current best practices for self-protection often are insufficient to counter advanced threats. Some general threats and vulnerabilities that frequently apply to UA systems will be briefly described. 1. CommunicationsUA systems often operate in environments that are denied, disconnected, intermittent, and limited (DDIL) (2). UA systems are typically designed with these limitations in mind and have countermeasures to handle planned degradations. In cases where the countermeasures are insufficient, attackers can adversely impact mission effectiveness. 2. Cyber Security Self-ProtectionUA systems typically use a variety of communications systems with external interfaces. Attackers can use a variety of tactics such as (3) to attempt to exploit software vulnerabilities and eventually gain access to sensitive data and software within the UA system. 3. Implementation DefectsUA systems often make use of complex software components that are difficult to rigorously assess. If attackers can get the UA system’s software to process maliciously formed inputs such as (4), then the attackers could take control of the software via well understood software exploitation techniques (5). Towards a Trustworthy Foundation for Assured Autonomous Systems STO-MP-IST-166 4 3 4. Machine Learning Subversion – Since some or all recognition decisions are tasked to subsystems within the UA system instead of to a human operator, these software components must be programmed to ‘recognize’ phenomena relevant to the system’s purpose (e.g., surveillance.) With the advent of recent machine learning research, real-time target recognition has become feasible. To protect this recognition software, we must address how to secure the different components of a modern machine learning system. 5. Ineffective AssessmentsSecurity assessments (e.g., penetration tests, Common Criteria) have traditionally focused on traditional information systems such as personal computer local area networks. As UA systems currently are heavily reliant on sensors and networks, techniques for assessing the risks associated with these dependencies must also evolve. If the military cannot adequately assess cyber risks associated with UA systems, then they risk overall mission failures due to an undetected cyber security failure within a mission critical UA system. In order for a UA system to maintain effectiveness throughout its life cycle, each of these vulnerabilities must be adequately addressed. In this paper, we briefly describe techniques that have a demonstrated level of effectiveness in countering these threats. Each technique is enhanced with a practical example. While these examples are informed by real-world implementations, they are not specifically related to any particular program known to the authors. 2.1 Example: A Notional UA System Throughout this paper, we will refer to a notional UA system in order to illustrate key concepts. This system will be an Unmanned Aerial Vehicle (UAV) with the primary mission of surveillance for high value targets using a camera. This vehicle will make use of a variety of passive sensors for input, a classifier program1 that analyses input from these sensors, radio frequency (RF) communications equipment for network connectivity between the UAV and the remote operators, and software that controls a camera based on inputs from the classifier2. Further, we will assume that a program office responsible for developing the UAV is planning this system as a major upgrade to an existing system that is deployed on a manned aircraft. While major components of the system will be new (e.g., the classifier, mission control), it must reuse major hardware and software components of the legacy system (e.g., passive sensor control, camera control) . This system must be designed and implemented in a manner that meets notional mission capabilities, but must also maintain system security despite a robust threat profile. Logically, the information flow for this system is depicted in Figure 2-1 Notional UAV Architecture. 1 For this paper we define a classifier as an algorithm with a data control map that classifies (