Internal Control Evaluation: A Computational Model of the Review Process

This study investigated the strategies by which experienced auditors evaluate systems of internal accounting controls. The research method included: (1) observations, using concurrent protocols, of a small sample of practicing auditors performing the internal control evaluation task; (2) extensive interviews with one of the practicing auditors; (3) formalization of auditor processes as a computational model; and (4) validation of the model. The simulation model was Im· plemented as an expert system and tuned to one auditor. The model output consists of a trace of the model processing including: (1) rec• ommendatlons for specific controls to be compliance tested; and (2) a list of management letter issues. The model was tested using new cases and cross-validated against the performance of additional auditors performing the task. A separate group of auditors rated the performance of the computational model and the additional auditors performing the validation task. RESEARCHERS, as well as tcach:rs and practitioners, arc interested in the processes that auditors use when making judgments and decisions. While expertise in such fields as medicine, physics, and chess has been studied intensely during the past 20 years (e.g., Klienmutz (1968), Einhorn [ 1980), Elstein et al. [ I 978) , and Johnson et al. (19821), comparatively little reRayman D. Meservy is Assistant Professor of Accounting and Inf ormation Systems, School of Urban and Public Affairs, Carnegie-Mellon University, Pittsburgh, PA 15213. Andrew D. Bailey, Jr. is Arthur Young Professor of Accounting, Faculty of Accounting and Management Inf ormaMeservy, Bailey and Johnson search has been done in the fields of business and management. The study and evaluation of internal accounting controls is a problem involving the expertise of well-trained auditors and is a requirement of every audit performed by CP As. This research examines auditor expertise as it relates to the audit task of evaluating internal accounting controls. This paper first focuses on expertise and the auditing task of reviewing and evaluating internal controls. The applicability of Decision Support Systems (DSS), Artificial Intelligence (Al), and Expert Systems (ES) to auditing tasks is then discussed. The current research covers the complete process of developing an expert system, but focuses most heavily on the cognitive science issues. The resulting computational model is intended to emulate the expert, i.e., the focus of our work is on understanding the expert's decision processes and not on building a better problem solver. The various methods of assessing the auditor's judgment processes used to build the knowledge base arc presented. An expert system shell was used to build a computational model of the resulting process. Validation of the resulting computational model using several approaches is discussed. EXT ANT AUDITING DSS/ Al/ES/ APPLICATIONS There arc many views about what should be included as DSSs and their relationship to Al and ES. While no definitive statements of definitions and characteristics for DDS exist, in this paper we use the term Decision Support Systems or DDS in a broad sense to refer to any interactive computer application that helps a decision maker by providing access to large data banks or by implementing a decision model, or both. Artificial Intelligence (Al) has been defined by Barr and Feigenbaum [1981 I as 45 "the part of computer science concerned with designing intelligent computer systems, that is, systems that exhibit the characteristics we associate with intelligence in human behavior understanding language, learning, reasoning, solving problems, and so on. " Expert systems have been defined by Stcfik ct al. [1982) as "problem-solving programs that solve substantial problems generally conceded as being difficult and requiring expertise. They arc called knowledge based because their performance depends critically on the use of facts and heuristics used by experts. " Feigenbaum (1978) says, "we must hypothesize from our experience to-date that the problem solving power exhibited in an intelligent agent's performance is primarily a consequence of the specialist's knowledge employed by the agent, and only very secondarily related to the generality and power of the inference method employed. Our agents must be knowledge rich, even if they arc methods poor. " Thus, expert systems attempt to capture specific knowledge from an acknowledged expert concerning a specific problem domain and to replicate the decision inference process used by this expert. In this paper, expert systems is used in lion Systems, College of Administrative Science, The Ohio State University, Columbus, OH 43210-1399. Paul E. Johnson is Professor of Management Science, Department of Management Science, School of Management, University of Minnesota, Minneapolis, MN55455. We would like to thank the Peat, Marwick & Mitchell Foundation, the McKnight Foundation, and the Artificial Intelligence Research Center of the University of Minnesota for their support. The paper represents the authors' beliefs and not those of the organizations noted above.