Automated Testing of Robotic and Cyberphysical Systems
Creators: name: Afzal, Afsoon
Publisher: name: Carnegie Mellon University
Year: 2021
Subjects: subject: 89999 Information and Computing Sciences not elsewhere classified
subject: FOS: Computer and information sciences

description: Robotics and cyberphysical systems are increasingly being deployed to settings where they are in frequent interaction with the public. Therefore, failures in these systems can be catastrophic by putting human lives in danger and causing extreme financial loss. Large-scale assessment of the quality of these systems before deployment can prevent these costly damages. Because of the complexity and other special features of these systems, testing, and more specifically automated testing, faces challenges. In this dissertation, I study the unique challenges of testing robotics and cyberphysical systems, and propose an end-to-end automated testing pipeline to provide tools and methods that can help roboticists in large-scale, automated testing of their systems. My key insight is that we can use (low-fidelity) simulation to automatically test robotic and cyberphysicalsystems, and identify many potentially catastrophic failures in advance at low cost. My core thesis is: Robotic and cyberphysical systems have unique features suchas interacting with the physical world and integrating hardware and software components, which creates challenges for automated, large-scale testing approaches. An automated testing framework using software-in-the-loop (low-fidelity) simulation can facilitate automated testing for these systems. This framework can be offered using a clustering approach as an automated oracle, and an evolutionary-based automated test input generation with scenario coverage fitness functions. To support this thesis, I conduct a number of qualitative, quantitative, and mixedmethod studies that 1) identify main challenges of testing robotic and cyberphysical systems, 2) show that low-fidelity simulation can be an effective approach in detectingbugs and errors with low cost, and 3) identify challenges of using simulators in automated testing. Additionally, I propose automated techniques for creating oracles and generatingtest inputs to facilitate automated testing of robotic and cyberphysical systems. I present an approach to automatically generate oracles for cyberphysical systems using clustering, which can observe and identify common patterns of system behavior. These patterns can be used to distinguish erroneous behavior of the system and actas an oracle. I evaluate the quality of test inputs for robotic systems with respect to their reliability, and effectiveness in revealing faults in the system. I observe a high rate of non-determinism among test executions that complicates test input generation and evaluation, and show that coverage-based metrics are generally poor indicators of test input quality. Finally, I present an evolutionary-based automated test generation approach with a fitness function that is based on scenario coverage. The automated oracle and automated test input generation approaches contribute to afully-automated testing framework that can perform large-scale, automated testing on robotic and cyberphysical systems in simulation.<br>