City Research Online Some conservative stopping rules for the operational testing of safety- critical software

Operational testing, which aims to generate sequences of test cases with the same statistical properties as those that would be experienced in real operational use, can be used to obtain quantitative measures of the reliability of software. In the case of safety critical software it is common to demand that all known faults are removed. This means that if there is a failure during the operational testing, the offending fault must be identified and removed. Thus an operational test for safety critical software takes the form of a specified number of test cases (or a specified period of working) that must be executed failurefree. This paper addresses the problem of specifying the numbers of test cases (or time periods) required for a test, when the previous test has terminated as a result of a failure. It has been proposed that, after the obligatory fix of the offending fault, the software should be treated as if it were completely novel, and be required to pass exactly the same test as originally specified. The reasoning here claims to be conservative, inasmuch as no credit is given for any previous failure-free operation prior to the failure that terminated the test. We show that, in fact, this is not a conservative approach in all cases, and propose instead some new Bayesian stopping rules. We show that the degree of conservatism in stopping rules depends upon the precise way in which the reliability requirement is expressed. We show that some rules are ‘completely’ conservative and argue that these are also precisely the ones that should be preferred on intuitive grounds. 1 Background and motivation The problem described here arose during recent discussions, in which one of the authors was involved, associated with the assessment of the software-based primary protection system of a nuclear reactor. The actual licensing process was based upon qualitative arguments, but the utility company volunteered to provide the regulator with a statistical demonstration that the system’s probability of failure upon demand (pfd) had achieved the design requirement1 of 10-3, independently of the licensing process. Here a ‘demand’ constitutes a set of circumstances, represented by sensor readings, that require the reactor to be shut down safely and kept thereafter in a safe state. The utility owned a simulator which it was agreed could produce input trajectories the demands that were statistically representative of those that the system would have to meet in real operational use. It was therefore required that the system be placed on test, be proffered 5000 demands, and show no failures. In the classical frequentist statistical framework, this ensures that 10-3 is (approximately) a 99% upper confidence bound on the true pfd [Parnas et al. 1991] (in fact 4603 perfectly executed demands gives a 99% bound); Bayesian analysis gives similar results [Miller et al. 1992, Littlewood & Strigini 1993]. The stopping rule here is clear: the demands will be executed until either a failure occurs, in which case the system has failed the test, or 5000 demands have been executed failure-free, in which case the system has passed the test. The problem arises in the event that a failure does occur. The regulator (and, indeed, the utility) will regard the system as not acceptable, and it would be necessary to remove the bug that has been revealed before considering the system as a candidate for a further test. Notice that, if the testing were allowed to continue (even without any attempt to remove the newly-found fault), a sufficient number of perfectly-executed demands might eventually be clocked up that even with the one failure within the total number executed we would still obtain a 99% upper confidence bound of 10-3 pfd. In fact a total of 6636 demands would be required. The stopping rule here arises mainly for non-statistical reasons: we take the reasonable conservative position that, in view of the safety-critical nature of the system, it could not be licensed for use whilst containing a known fault, whatever the evidence that it nevertheless met the pfd requirement. The regulator thus insists on being shown evidence that any fault found in testing has been removed, and then argues that the only positive evidence in favour of the system that he/she will accept from testing is the amount of perfect working since the last faultremoval. In our discussions of this testing scenario, the question arose: what new testing requirement should be imposed upon the system following a failure of this well-defined test? In particular, is it reasonable to require that the new version of the software be given exactly the same hurdle to overcome as was initially devised? The intuitive objection to such an approach is that it ignores the fact that we have received some information from the first test; in particular, in the event that the failure that terminated the test occurred after executing only a small number of demands, we have received some bad news. A regulator might reasonably be wary of ignoring such information and merely requiring the same target be achieved from the new test. The fact 1 The comparative modesty of this requirement arises as a result of there being several other lines of defence against unacceptable accidents most notably, a simple, non-software-based, secondary system. It should also be noted that this testing formed only a part of the evaluation of this software-based protection system, and did not form part of the licensing process. that a failure has been observed early in test may, for example, be evidence of low software quality, as a result of poor production practices, or of unacceptable characteristics of the system (for example, excessive complexity). Such a wariness seems reasonable when we note that, even if discovered faults were never fixed, and regardless of the true probability of failure on demand, if we repeatedly put the system through tests comprising 5000 statistically representative demands, eventually it will succeed in passing a test. It seems unlikely that the regulator will in reality have the luxury of demanding of the utility that it scrap the existing system and start again if we could take this course, it might be reasonable to suggest that the new system should be subjected to the same acceptance criterion as the first2. What is needed, therefore, is a modification of the above simple stopping rule that allows failures to occur during test, and consequent removal of the faults that these reveal, but which takes into account this potential ‘bad news’ in specifying the further number of failurefree demands that must be executed. The intention here is to devise stopping rules that retain the flavour of the one described above. That is, we require a stopping rule that allows the parties at any time during a test to agree that ‘if the system executes n further demands without failure it will be deemed to have passed the test, otherwise testing must continue. ’ More formally, we require a rule of the following kind: 1 At the start of test, we compute the number, n1, of demands that must be executed failure-free for the test to succeed and stop. 2 The system is put on test and either successfully executes the n1 demands, in which case the test stops and the system is declared to have achieved its pfd requirement, or a failure is observed on demand s1 (<n1), in which case the test is stopped. 3 In the light of the evidence of one failure in s1 demands, we compute the number, n2, of further demands that must be executed failure-free for the next test to succeed and stop. 4 The system is put on test again and either successfully executes the n2 demands, in which case the test stops and the system is declared to have achieved its pfd requirement, or a failure is observed on demand s1 + s2 (s2<n2), in which case this test is stopped. And so on. Clearly, it is not certain that in practice the process will terminate. This is in accord with intuition, since termination implies that the system is finally acceptable and this may never 2 Although even here it might be argued that there should be some carry-over of evidence. For example, if the same development team were used, we might think that their failure to make the first system pass the test was evidence that they were not greatly competent, and this might depress our expectations of their likely success on their second try. Equally, the first failure might make us believe more strongly that the problem being tackled in building the system is a ‘difficult’ one, and that any putative solution is thus more likely to fail. be the case. Essentially what is happening is a competition between the ‘good news’ represented by the most recent failure-free executions, and the information coming from the accumulating failures. In the next section we present simple Bayesian solutions to this problem which retain some of the conservatism that a regulator might desire. This deals with the problem in the context of a demand-based system, as above; Section 3 treats the similar problem concerning the testing of a system that is required to operate continuously. 2 Stopping rules for demand-based systems We shall assume that successive demands are statistically independent Bernoulli trials. Let p be the probability of failure on demand. Thus, given p, the number of failures in n demands, R, has a Binomial distribution: P(R = r)=nCr p r (1 − p) (1) and in particular P(R = 0) = (1 − p) (2) Within the Bayesian framework we represent our a priori knowledge about the parameter of interest, here p, by the prior distribution. There are advantages in using a prior distribution from the conjugate family3 which in this case is the Beta(a,b) distribution: f ( p) = p (1 − p) B(a,b) (3) where B(a,b) is the Beta function and a>0, b>0 are chosen by ‘you’ to represent ‘your’ belief about p prior to seeing any test results. In some cases it might be possible to use information about the system an