Using Metamorphic Testing at Runtime to Detect Defects in Applications without Test Oracles

Assuring the quality of applications such as those in the fields of scientific calculations, simulations, optimizations, data mining, machine learning, etc. presents a challenge because conventional software testing processes do not always apply: in particular, it is difficult to detect subtle errors, faults, defects or anomalies in many applications in these domains because there is no reliable “test oracle” to indicate what the correct output should be for arbitrary input. The general class of software systems with no reliable test oracle available is sometimes known as “non-testable programs”. Although knowing the correct output of such non-testable programs is impossible for arbitrary input, we have observed that even when there is no oracle in the general case, there can still be a limited subset of inputs for which the output can, in fact, be predicted. These are typically only very simple inputs, however, and may not have much power at revealing defects. Other inputs that, for instance, push boundary or timing limits can at least reveal gross errors, e.g., catastrophic failures (crashes), but we require an approach that will reveal more subtle defects for arbitrary input. Without an oracle, it is impossible to know what the expected output should be for any given input, but it may be possible to predict how changes to the input should cause changes to the output, and thus draw a relation between a set of inputs and the set of their respective outputs. We propose to build on an approach called “metamorphic testing”, which takes advantage of properties of functions such that it is possible to predict expected changes to the output for particular changes to the input. If the change is not as expected, then a defect must exist. In order to generate the test cases, metamorphic testing requires the initial input and output values, which could be generated using techniques like equivalence partitioning or random testing. However, inputs chosen using these techniques might miss some defects, since they might not happen to consider a sufficient variety of potential system states. Some defects in such systems may only be found under certain application states or for certain inputs that may not have been tested. Thus, we require a strategy that specifically considers these field states, by using real inputs and outputs from actual executions rather than just those generated in the testing lab. Our approach, therefore, entails continuing metamorphic testing of the application as it runs in the deployment environment. For either an individual function or for the entire application, we first capture initial input/output pairs that are taken from actual executions in the field. Although we cannot know whether the output is correct (since there is no general test oracle), we at least know that the input is something that comes up in practice, and is thus useful as a valid test case. We then apply the function’s or application’s “metamorphic properties” to derive new test input, so that we should be able to predict the corresponding test output. Although we cannot know whether the test output is correct either, if it is not as predicted then there is a defect. Since this process is conducted in the field, we must ensure that users do not notice this testing, e.g., see the test output, experience a sudden performance lag, etc. The principal hypothesis is that, for programs that do not have a test oracle, conducting metamorphic testing within the context of the application running in the field can reveal defects that would not ordinarily otherwise be found. Additionally, we believe that this can be done without affecting the application state from the users’ perspective, and with acceptable performance overhead. By way of proving these hypotheses, this thesis will make three contributions. First, we will present an approach called Automated Metamorphic System Testing. This will involve automating system-level metamorphic testing by treating the application as a black box and checking that the metamorphic properties of the entire application hold after execution. This will allow for metamorphic testing to be conducted in the production environment without affecting the user, and will not require the tester to have access to the source code. The tests do