Reflections on ratings

Traditional computer security ratings have focused on determining the existence of a single configuration of a given system that meets the standards of a particular gradation of the rating. The author sees the use of such ratings in isolation from real world usage data leading to a situation where system security ratings bear little or no resemblance to the level of security that may be expected in actual use. Anecdotal evidence indicates that while the configuration which passed the ratings may represent a peak, the drop off from that peak down to the typical level over a wide range of configurations is often precipitously steep. It is possible that by concentrating more attention on ratings which more accurately reflect typical usage, we may see solutions developed which may rise to less lofty peaks, but which have a far higher average level of security under real usage patterns. 1 History and Present Conditions Over the years, there have been many ratings, criteria, assurance levels, capability maturity models, etc. designed to provide insight into the security of computing systems. These schemes have had some notable differences. For example, if one were to compare and contrast the Trusted Computer System Evaluation Criteria, also known as "The Orange Book" with the International Common Criteria system that is currently being espoused, one would find noticeable differences. The Orange Book sets down a limited number of discrete ratings, while the CC provides a language which can be used to specify an almost infinite number of possible configurations and profiles, with emphasis on all sorts of different aspect of the problem. The Orange Book couples security functions and assurance measures, in the belief that the best mechanism in the world is useless if you have no assurance that it was adequately constructed and it is properly invoked. Similarly, high assurance is useless if it provides confidence that an inadequate mechanism has been constructed and will be invoked. The Orange Book sets out combinations of assurance and mechanism, in the desire to balance the protection provided by each. The Common Criteria, however, decouptes function and assurance, allowing one to create, if one wishes, a profile for a theoretically strong security system design with absolutely no assurance that the stated features have actually been designed as specified, or function as advertised. This paper is authored by an employee of the U.S. Government and is in the public domain. New Security Paradigm Workshop 9/00 Ballycotton, Co. Cork, Ireland ACM ISBN 1-58113o260-3/01/0002 The Orange Book was conceived as part of a system where a small group of U.S. Government evaluators would examine systems destined for U.S. government use, and rule on their place in the Digraph hierarchy. The CC, by contrast, is geared toward creating a commercial security industry. In the industry envisioned, anybody can create a profile. Then, any manufacturer can build a system to match that profile. Given the system and the profile, anybody who cares to pay a testing lab for their services can have an evaluation done comparing the system against the profile, The Orange Book, and most of the rest of the "Rainbow Series" was developed with stand-alone, centralized systems in mind. While there was a "Trusted Network Interpretation", the interpretation was in some senses a permutation of the central assumptions of the Orange Book. In contrast, the Common Criteria allows one the freedom to specify a secure kangaroo if one wishes, provided one uses the proper terminology, and can adequately state the threats, goals, mechanisms, procedures and countermeasures which apply to the kangaroo security domain. It would seem that given the marked contrast between these two approaches, one could come to several conclusions about the idea of security ratings for systems. One might theorize that there has been an evolution from the Orange Book to the Common Criteria. One might view them as opposite ends of a pendulum swing, with the current practice at any point in the future predicted to fall somewhere along the spectrum. One might view them as different business models, and debate the merits of each, perhaps using analogies and contrasts to the history of the Orange book to attempt predict the economic success and market acceptance of the Common Criteria. In summary, the contrast between the Orange Book and the Common Criteria seems significant. One could probably find similar dissimilarities between either of these schemes and any of the other schemes that have been espoused over the years. The one detail that would probably be overlooked is that in a critical way, they are all variants of the s a m e s c h e m e . One might ask how that could be, given all the fundamental differences in structure and philosophy noted previously, and assuming that similar fundamental differences separate all the