Knowledge based systems that have some idea of their limits

A major problem with Knowledge Based Systems (KBS) is that they do not know when they have reached the limits of their knowledge and so are likely to make stupid conclusions. One possible approach to this is have the KBS keep some sort of record of every type of case it has seen and issue a warning when it is dealing with a case outside its range of experience. In these studies we have used the simple technique of issuing a warning if any attribute has a value outside the range of those seen previously for cases satisfying the same sequence of rules during inference. This technique has been evaluated using standard machine learning data bases and a simulated expert (built by machine learning) rather than a human expert. The conclusions from this study are that single attribute warnings cannot be guaranteed to pick up all potential errors, but go a long way towards a knowledge acquisition methodology whereby a KBS will fairly reliably prompt that it needs more knowledge INTRODUCTION A major problem with expert systems is that they are "brittle". That is, the expert system does not realise when its knowledge is inadequate and that advice should be sought from some more expert source. It is a well known phenomenon with KBS that they may combine very high levels of expertise with a lack of common sense. They may outperform a trainee, but will not have the trainee's insight that there may be a problem beyond their competence. The classic (apocryphal? ) example is the "pregnant male syndrome" where the expert system deduces the patient is pregnant because of laboratory results but has never been informed that only females become pregnant and so erroneously deduces that a male patient is pregnant. The CYC project is one attempt at a solution to this problem (Lenat & Feigenbaum 1991). CYC is an attempt to build a knowledge base of sufficient consensus or common sense knowledge so that specific KBS built on top of CYC will be able to fall back on common sense in a human like fashion and avoid stupid errors. CYC is aimed specifically at errors like the pregnant male syndrome where common sense if it was available would help the system reach a correct conclusion. Apart from the problems of common sense that arise, there are also issues of brittleness within the domain itself. Even with apparently fairly complete systems, stupid errors are liable to be made. For an example see (Compton, et al. 1989). Despite the availability of techniques to check if a knowledge base is complete(Preece, et al. 1992), in practice this is not very useful. Most interesting practical systems cover a huge space of possible data patterns, so that an expert will not be asked about all of them in building a system. The vast majority of these data patterns may never occur in practice (Colomb & Sienkiewicz 1996) and experience in dealing with experts suggests that their responses become rather speculative when dealing with data patterns outside their experience (Compton & Jansen 1990). As will be discussed below we have been developing approaches to knowledge acquisition based on correcting errors as they occur, so that the knowledge base is developed whilst the system is in actual use and errors are detected. Clearly for such a system it would be very attractive if the system knew when it was dealing with situations beyond its present competence and was able to request further knowledge acquisition. However, it should be noted that such a feature is perhaps even more desirable for conventional systems. These are systems where we have attempted to make the knowledge as complete as possible before the system is put into use. Although a high level of performance is probably verified before the system is deployed, all this means is that the system is likely to make few errors; but what these are, when they will occur and their significance is generally not known. We are still a long way from fully verified and validated systems. The aim then of the present work is to develop an approach whereby a KBS will realise that a case it is dealing with is beyond its competence and will request help. For a system with an apparently high level of performance this will be an occasional warning that the system's output needs checking. With a system designed for incremental development whilst in actual use, the system will be responsible for requesting knowledge acquisition as needed. We have previously attempted to use to exploit the particular structure of the incremental knowledge acquisition method, Ripple Down Rules (RDR), to suggest when a conclusion for a case is incorrect (Kang & Compton 1992). This approach was based on the idea that the KBS process a case and reaches a conclusion, but that the system's knowledge is checked to see if knowledge in another context, not reached by the case, may actually satisfy the case and thereby suggest that the conclusion arrived at may be wrong. The motivation was to capture the human notion that a certain conclusion seems appropriate, but that there is just something about the case that reminds us of some other quite different context so we have some doubt about the more immediately obvious conclusion. Although this approach is useful and has some interesting cognitive features and is being developed further, it depends on other knowledge already in the knowledge base and so does not satisfy the requirement for a system that will warn when a case is outside the system's experience. The approach we have now developed and which is evaluated here is based simply on the idea of keeping a sufficient record of all cases seen by the system and warning about any cases different from those previously seen. We use the term "prudence" to describe this behaviour of issuing a warning every time a case is in some way unusual. We have previously evaluated this approach by rebuilding 600 rules of a medical expert system by hand (Edwards, et al. 1995). The results of this initial study were very promising in that as the knowledge based was developed, every error the system made, that is every case where further knowledge acquisition was required was correctly warned about. That is the system correctly prompted for all knowledge acquisition. However, the warnings were unnecessary in some 40% of cases. This seemed to be a major advance in that normally such a system would required all output to be checked whereas with the prudence feature included only 40% of cases had to be checked. More importantlyno undetected errors were made. In the work described here we have carried out a larger scale evaluation based on using a simulated expert for knowledge acquisition enabling a number of studies based on different domains. The essential idea is that using a simulated expert, despite the limitations and non-human nature of its expertise, enables the performance of a system to be evaluated on large data sets, where this would be very difficult to organise with human experts. We have previously used this approach to evaluate the efficiency of the RDR methodology (Compton, et al. 1995). The strategy of warning every time a case is different from those previously seen by the system is very simple. However, it overcomes the problem of dealing with parts of the state space where cases do not occur or occur extremely rarely. With the approach of providing warnings one does not need to develop rules for these parts of the state space; the rules are adequate for the data patterns that generally occur and for the rare occasions when a novel pattern occurs these are warned about so that other more expert advice can be sought. METHODS Recognising Unusual Cases We identify cases that are unusual with respect to the particular classification being made and with respect to the particular sequence of rules that were satisfied reaching the classification for that case. We distinguish between rule path and the actual conclusion or classification as the same conclusion may be given by a number of rules. All the data in these studies is in attribute-value form. Associated with each rule and also with each conclusion we keep a set of the values seen for each attribute. If the value for an attribute for a case being processed is not already included in the list of values for that attribute a warning is given to check the conclusion for the case. If the conclusion is then accepted, the list of values for the attribute stored with that rule are updated with the new value. If the conclusion is wrong and new a rule is added, the values for the attributes for this particular case become the initial set of acceptable values for the attributes associated with the new rule. Numerical data is taken as a special case and instead of keeping the actual values, the range of values seen is kept so that a value for an attribute in a case may a new maximum or minimum. The data requires some consideration before this technique can be used and any datum that is unique to the case, such as an identification number, needs to be removed. Otherwise, all cases will be warned about. Apart from such data all other attributes are considered; both attributes that are known to be relevant to the problem domain and those which may not be so relevant. One assumes that the data that constitute the case are broadly relevant to the task otherwise they would not have been collected, but not all the data will be immediately relevant to the conclusions under consideration. For example, in a medical KBS one may be building a KBS for a particular diagnostic sub-domain. There will almost certainly be other laboratory and clinical information available, which although not apparently relevant to the domain, may be relevant under some circumstances. A limitation of the method chosen is that we are implicitly assuming that all attributes are independent and so the values can be considered independently. As will be seen this is a good first approximation, but is not appropriate if one requires a system guarant