CONSTRUCT VALIDITY OF COMPUTER BASED TESTS

One of the major issues in the field of computer based testing and assessment has to deal with the issue of does a conventional test change when it is reformatted for a computer based presentation even if all of the items and the test itself appear to be identical. One researcher (Green, 1988) addressed these issues primarily in his interpretation whether the construct validity of the test changed from a conventional paper pencil administration to one where it is administered totally on the computer. Some of the main issues addressed had to deal with the following characteristics which may affect the construct validity of the computer based administration. They are: a) Passive omitting b) Back tracking; c) Screen capacity; d) Graphics; e) Responding; f) Time limits: and g) Adaptive tests and related dimensionality. If even one on the topics to be addressed changes when a test is administered with a computer then the tests' prior norms and validity may have to be re-established in its new format.

**Passive omitting.** On a paper pencil test a respondent can pass on one or two items (for example items two and three) and then he or she can respond to item four and then item five. In fact a respondent can review the whole test before they go back to start answering and filling in responses to questions. In a computer based test (CBT) this cannot be done unless another choice command or control function keys are provided to allow for a "skip"or "next"item" pass etc. Even if this "skip" and "return" function is allowed it places the examinee in a different mental set and it also requires a breakdown of attention to the task on hand (responding to the cognitive nature of the material being evaluated) to mentally rearranging response patterns through different keyboard manipulations.

**Back tracking.** This occurs when an item has been previously skipped or passed as in passive omitting above or when a student answers a later question (item 10 for example) and now realises that he or she had made a mistake in a prior item (item 3 for example) and that the answer cannot be changed or can only be changed by further mechanical manipulations of the keyboard and the related user software.

**Screen capacity.** Prior research by human factor specialists (Sandals 1987) state that approximately 64% of the computer screen should be blank when information is presented in a learning or testing situation. Thus there is a chance that some items such as those that include a lot of reading comprehension may not fit on one screen and actually may take up two or three screens before a response can be made. In the paper version all of the information may be included on an 8-1/2 x 11 page.

**Graphics.** Some of the same issues raised in C above also relates to the size of the screen. Unless the user is using a screen with high resolution colour graphics (such as super VGA) or digitally stored images or laser discs or CD-ROMS then there will be difficulty in presenting many graphics in the same resolution as the original in the printed test booklet. The technology is available to make the reproduction almost 100% accurate, the limitation is the
related high costs for many educational institutions on affording this sophisticated state of the art hardware and software.

*Responding.* The response in our computer based test usually consists of pressing a key and in most cases this is faster than transferring an answer to an answer sheet and thus this can cause a difference in scores with highly speeded components on tests that may cause vigilance error in the filling in of the answer sheets as reported by (Sandals, 1970). Thus responding may be faster and more accurate through a keyboard, mouse, a light pen, and also the computer may not accept an incorrect answer if it is not in a proper field and thus, as a consequence feedback is given. However, feedback for a misplaced response cannot be provided in a paper pencil test. Thus the whole process of responding may affect the overall test score and the construct validity especially in a speeded test.

*Time limits.* In most grouped test situations a time limit is given in order to allow a teacher or tester control of the testing situation for the norm of the group. However, in the case of the computer the question is raised whether the customary time limits should be abandoned unless the test has a speeded component which is central to the construct validity of the instrument. Thus the construct validity may change if the computer administration does not have the time limit of the paper pencil version.

*Adaptive testing and dimensionality.* The major construct validity problem with computer adapted tests (CAT) is that the computer constantly changes the test and the item selections based on the prior response or the prior group of responses. Thus passive omitting is not possible, neither is back tracking or the changing of a prior response. In (CAT) a test item or pattern cannot be changed once an item response has been made or skipped. In addition, the dimensionality and content validity may change since usually no two students get the same test. Thus the usual criterion referenced test decisions can be made but norm referenced comparisons become impossible to report with traditional reporting methods. Two students may go through the exam with one taking only half as many items as another with the items that are actually the same being in the 20% range. The usual interpretation of test results may change since a direct item to item comparison may not take place only the domain can be cross validated. It may become more difficult to compare student performance when the domain being tested is in Language Arts and Social Studies in comparison to Math and Science where the concepts are more hierarchical and well defined. Thus once a conventional test is placed in a (CAT) mode the construct validity may change significantly as does the content validity and probably most of the reliability of the original test.

Thus these construct validity issues really question whether the computer administration of a conventional test is really measuring the same original and if not, new norms have to be provided in addition to the new interpretation of the results from the (CBT or CAT).
CONCLUSIONS

This paper has presented an overview of the role computer based assess-
ment and diagnosis has played in both educational and psychological environ-
ments. Many who made great predictions in the early 1970s for computer based
testing were over optimistic on both the acceptance, funds, research and
availability of the hardware and software for the mid 1980s. It is only now that
we are seeing the reduction in costs and the research in psychometric theory
and expert systems that are needed to make wide ranging applications of
computer based assessment and diagnosis a reality. The advent of interactive
cd's and CD-ROM's are now going to allow us to provide verbal instructions and
graphics and pictures that provide a realistic alternative to conventional
individualized assessment instruments. Again the whole issue of the use and
misuse of computers in education will come into play if those in power make
some of the same mistakes that computers educators did from 19751983. In
addition, if the concerns outlined in Table 1 are readdressed then there are
many potential benefits for society in general in the use of computer based
assessment and diagnosis.

REFERENCES

Bunderson, C., Inouye, D., & Olsen, J. (1989). The four generations of computerized education measurement. In Robert Linn (Ed.), Educational measurement, (3rd edition), 367-407. NY: American Council of Education/ MacMillan.

Colbourn, M. J., & McLeod, J. (1983). The potential and feasibility of computer-guided educational diagnosis. In R.E.A. Mason (Ed.), *Information processing*, 83. North-Holland: Elsevier Science Publishers.

Dubois, PH. (1970). A history of psychological testing. Boston: Allyn & Bacon.

Eaves, C. R. (1984-1985). Educational assessment in the United States, *Diagnostique* **10**, *5-39*.

Fifield, M. B. (1989) Psychoeducational testing and the personal computer. *Journal of Special Education Technology, 9*(3), 136-143.

Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation on student achievement:Ameta-analysis. *Exceptional Children*, *53*,199-208.

Fuchs, L. S. (1987). Curriculum-based measurement for instructional program development. *Teaching Exceptional Children*, 20(1), 42-44.

Fuchs, L. S. (1988). Developing computer-managed instruction on teacher's implementation of systematic monitoring programs and student achievement. *Journal of Educational Research, 81*,294-304.

Fuchs, L. S., & Fuchs, D. (1989). Enhancing curriculum-based measurement through computer applications: Review of research and practice. *School Psychology Review, 18*, 318-327.

Greaud, V., & Green, B.F. (1986) Equivalence of conventional and computer presentation of speed teats. *Applied Psychological Measurement, 10*, 23-24.
Green, B. F. (1988). Construct validity of computer-based tests. In H. Wainer, & H.I. Braun (Eds.), *Test validity*. Hillsdale, N.J.: Erlbaum.

Jackson, D. N. (1986). *Computer based personality testing*. Washington D.C.: APA.

Knights, R. M., Richardson, D. H., & McNarry, L. D. (1973) Automated vs clinical administration of the Peabody Picture Vocabulary Test and the Coloured Progressive Matrices. *American Board of Mental Deficiency, 78*, 223-225.

Olsen, J. B., Maynes, D. D., Slawson, D., & Ho, K. (1989). Comparisons of paper-administered, computer-administered and computerized adaptive achievement tests. *Journal of Educational Computing Research, 5*(3), 311-326.

Poteet, J. A., & Eaves, R. C. (Eds.). (1984-1985). Monograph: Perspectives in special education assessment [Special Issue]. *Diagnostique*, 10(1-4).

Sandals, L. H. (1970). *Vigilance errors on a search examination*. Unpublished masters thesis, Xavier University, Cincinnati.

Sandals, L. H. (1987). The role of screen design graphics, colour and sound in computer based learning. How much is too much or too little. *Proceedings of the International Conference on Computer Assisted Learning in Post Secondary Education*. (pgs. 189-196). Calgary, Alberta, Canada: University of Calgary.

## AUTHOR

Lauran H. Sandals is a Professor in the Department of Educational Psychology at the University of Calgary, Education Tower, Calgary, Alberta T2N 1N4.
Book Reviews

Mary Kennedy, Editor

*The Design, Development and Evaluation of Instructional Software* by Michael J. Hannafin and Kyle L. Peck. New York: Macmillan Publishing Co., 1988.

Reviewed by L.F. (Len) Proctor

Purpose of the Book

This book suggests that the quantity and quality of software must be improved if computers are to have a positive impact on the field of education. In Hannafin's and Peck's view, it is the teacher's skill, knowledge of design and commitment to quality that are the most important factors in the creation of high-quality, computer-based instruction. As a result, they have emphasized the instructional design process and not the hardware or software used in lesson creation. Novice authors who use the recommendations and suggestions outlined in this publication to guide their software development will avoid many of the pitfalls often associated with low-quality computer-baaed instruction.

Structure of the Book

This book contains seven sections which have been subdivided into twenty two chapters. Each chapter contains a list of objectives, an extensive reference list, suggestions for related reading and review exercises. The review exercises have been designed to reinforce the concepts and principles presented in the body of each chapter.

The first three sections provide a pedagogical basis for the development of computer-based instruction. Section One contains a brief overview of CAI and a description of the characteristics of effective CAI. Section Two presents a discussion of how to combine the strengths of teachers and computers to produce powerful teaching systems. Next is a description of a generic instruc-
tional design model which in turn serves as a general organizer for the remainder of the text. A good ID model is important because the success or failure of any CAI lesson depends more on the design of the lesson than on any other single element in the authoring process. Section Three completes the treatment of the pedagogical basis for the development of CBI by providing a structure for assessing the viability of CAI as a solution to an educational problem, a method of task analysis, and a strategy for the construction of learning objectives.

The mechanics of developing and producing computer-based instruction begins in Section Four. Section Four covers the topics of flowcharting, branching, mode selection (e.g., tutorial), frame layout, emphasis options (e.g., type size), interactivity, and student response management. Section Five focusses on the differences between print-based lessons and computer-based lessons. This section describes some of the languages available for CBI lesson creation, lesson organization and how to collect learner response data. Section Six completes the description of the instructional design model by detailing how to carry out the evaluation and revision of each component of the development process. This section is particularly valuable because the evaluation criteria presented here apply equally well to internally and externally developed CAI instruction.

The last section of the book considers both the present and possible future status of computer-based instruction. Section Seven describes peripherals that may be added to the system, interactive multi-media, intelligent CAI, and the emergence of computer networks. The book concludes with a discussion of factors currently influencing the role of CAI, a glossary of terms, author and subject indexes, and a list of recommended readings to guide authors in extending their study of the topics presented.

## Critique

For the beginning author, the planning, organizing and production of CAI resources can become a very complex task. The mere act of trying to decide where to begin can often lead to confusion. Hannafin's and Peck's book succeeds in reducing the complexity of the authoring process to manageable limits. It is well organized, clearly written and substantially referenced. While the instructional design model presented is not as extensive as models found in other sources, the essential elements of the design process have been extracted and customized in order to accommodate the needs of a beginning author. The book provides a good framework for making lesson design decisions. Hannafin and Peck assume that novice CBI authors are competent teachers and knowledgeable, experienced computer users. For those authors who do not have these entry level skills, they offer an alternative. They suggest that the user who has little or no programming expertise or design experience could use prestructured templates to create computer-based instruction.

For the novice author who is willing to adopt a linear "programmed" learning approach to CAI development this would probably be all right. But,
for authors who wish to use more complex interactions or make use of student tracking capabilities to control lesson presentation elements, there is no substitute for knowing about variables, functions and program control structures. Second, a basic knowledge of the instructional design process would be of help to any author who finds it pedagogically desirable to deviate from the lesson plan presented in a prestructured template.

The suggestions and recommendations made by Hannafin and Peck for implementing learner control, the use of navigation aids, screen design, and the management of student responses are well defined and presented. However, the guidelines given for presenting feedback to students are minimal. They only mention that “feedback frames are used to provide students with the correctness, incorrectness or quality of their responses,” Only one example of incorrect response feedback was given. Confirmation, correct response, explanatory and bug-related examples of feedback could have also been given. This is one topic the authors could have treated in greater depth.

One of the most useful chapters in the book is the chapter on evaluating CAI lessons. Hannafin and Peck define evaluation as an “ongoing process used to determine whether lesson objectives have been met, to identify the reasons for the observed performance, and to identify those portions of a lesson where modifications are required.” They have chosen not to stress elaborate statistical methods for gathering empirical evidence. Instead, they have developed a series of checklists which serve as systematic guides to gathering informal, anecdotal types of data. Each checklist highlights a series of key points to be considered in the evaluation of the lesson. For example, in the area of instructional adequacy Hannafin and Peck ask: “Are the directions for lesson control clearly stated?” In this case, while the directions are not numerically quantified, it is easy to see that if students have trouble navigating through the lesson they may become easily discouraged or frustrated with the lesson.

This example highlights one problem inherent in summarizing data from this type of evaluation procedure. Even though the lesson may crash in certain circumstances, it may still 'pass' its evaluation. The problem here is that highly rated assessment, despite a fatal flaw such as "crashing," may mislead the evaluator into drawing positive conclusions about the lesson. To avoid this potentially disastrous outcome, Hannafin and Peck recommend the use of a combination of assessment criteria and fatal flaw criteria for anecdotal methods of CAI lesson evaluation. When both these components are considered concurrently, evaluators are less likely to become the victims of their own evaluation system.

In conclusion, I suggest that this book would be a good starting point for anyone who is seriously considering the development of computer-based instruction. It is not an authoring system specific handbook; it is a generic lesson authoring guide. It does not deal extensively with topics such as intelligent CAI systems, but it does offer a masterful introduction to frame-based approaches to CBI. Perhaps other topics such as adaptive instructional
designs, expert systems and artificial intelligence will be included in a revised edition or a future companion publication.

REVIEWER

L.F. (Len) Proctor is an Associate Professor in the Department of Communications, Continuing and Vocational Education, University of Saskatchewan, Saskatoon, SK S7N OWO.

*Evaluating Open and Distance Learning* by Mary Thorpe. Mississauga, ON: Copp, Clark, Pitman, 1988. ISBN 0-582-90119-7 (CDN $29.95)

*Reviewed by Mary F. Kennedy*

According to the author, this book is written for the practitioner — not the evaluation practitioner, but the practitioner in the field of open/distance learning. Unlike most books on evaluation, it does not include an historical overview of evaluation theory and design. Rather it goes to the heart of the matter — evaluation as a practice.

*Evaluating Open and Distance Learning* is divided into three sections. Part One: *Open Learning and Distance Education* presents two brief chapters on evaluation, defining the term and setting the focus on who uses and/or ultimately benefits from evaluation. Thorpe makes a case for learner collaboration and a team approach. A really nice feature of Chapter Two is the inclusion of five case studies, all selected on the basis of interest and application to the potential audience.

Part Two: *Evaluation in Practice* contains four chapters on learner self-evaluation, tuition or tutoring, counselling, and course or learning materials. Each chapter provides an overview of evaluation activity in relation to that specific component of open and distance learning. The activities, and Thorpe's suggestions regarding implementing evaluation, draw on data from past completed evaluations. There appears to be, on cursory reading, an over-reliance on Open University data, but, as Thorpe explains, there is little evidence that evaluation is being implemented elsewhere in open and distance education settings. Numerous samples of evaluation checklists, survey instruments, and interview guides are included in these chapters, providing the reader with a flavour of the type of evaluation activity undertaken.

Part Three: *The Process of Data Collection* includes two chapters and a conclusion. As the author indicates, the purpose of this section is not to provide a how-to manual, but to focus on the application of these methods to open and distance learning. Thorpe reiterates the view that learners and practitioners should define the type of evaluation they want, since evaluation should be
undertaken for the development of learning and the enrichment of learning experiences.

The introductory chapter of this section - Chapter Seven - is the weakest of the book. The problem lies in the scope of material included. Entitled Planning and Data Collection, it is supposedly presented within the context of the social organization of evaluation, but this concept gets lost early on in the nitty-gritty of topics such as chairing meetings, and keeping minutes and ensuring equality of contribution from evaluation team members. In addition there is unevenness of treatment of the various data collection techniques, and so little information on them as to render the treatment meaningless.

Chapter Eight, the final chapter, focuses on the social implications of evaluation, an area Thorpe claims is underemphasized in most evaluation texts. On that point there is no argument. She draws together the elements of this rather complex social process called evaluation, and while not as prescriptive as in earlier sections, the organization and presentation of the information is helpful to the reader who is interested in looking beyond the doing of evaluation activity to the implementation of practices based on evaluation findings.

Her conclusion, while brief, deals with the issue of quality control and she makes the plea for moving beyond the attainment of an acceptable standard of program implementation, the aim of most quality control policies, toward excellence.

*Evaluating Open and Distance Learning* is a good book for its intended audience. Little understanding of or background in evaluation is required to implement evaluation activity on the scale recommended by the author and the book would certainly provide basic guidance in getting started. One would have to accept the notion, of course, that any type of evaluation is an improvement over no evaluation.

The book has certain bonus features which would make it popular with the novice evaluator. The organization is appealing, and the discrete chapters on each major component to be considered in the evaluation of open and distance education provides the reader with an idea of the scope of course evaluations. Each chapter is followed by a short but good selection of further readings which Thorpe considerably annotates for the reader. The focus of evaluation activity is not research-based; rather, the emphasis is always practical. Formative evaluation is the thrust, for the purpose of course or program monitoring, modification and improvement.

In summary, for the clearly defined audience of open and distance education practitioners this book is ideal as an easy-reading, non-threatening introduction to evaluation, and many practitioners in that field would fit the category of novice evaluator. For those already versed in program evaluation, the book has little to offer.
REVIEWER

Mary F. Kennedy is Associate Professor of Educational Technology at Memorial University of Newfoundland. She is also currently Book Review Editor for *CJEC*.
## Information for Authors

CJEC welcomes papers on all aspects of educational communication and technology. Topics include, but are not limited to: media and computer applications in education, learning resource centers, communication and instructional theory, instructional design, simulation, gaming and other aspects of the use of technology in the learning process. These may take the form of reviews of literature, descriptions of approaches or procedures, descriptions of new applications, theoretical discussions and reports of research,

## Manuscript Categories

Manuscripts may fall into one of two classes: *Genera*, dealing with a topic or issue at a general level (although reference to specific instances or examples may be included), and *Profiles*, dealing with or describing only a specific instance of an approach, technique, program, project, etc. A Profile may be thought of as a descriptive case study.

Most manuscripts dealing with a topic in general should include reference to supportive literature, while manuscripts submitted to the Profile category may or may not. The Editor reserves the right to change the designation of a manuscript or to make a designation, if none has been made previously by the author. Authors interested in determining the suitability of materials should consult past issues of CJEC or contact the Editor.

All manuscripts received by the Editor (either general or profile) will be judged for suitability, contribution, accuracy, etc. by a panel of anonymous reviewers designated at the time of submission. Normally, the review process requires about eight weeks. There are no deadlines for the submission of manuscripts.

## Manuscript Preparation

Manuscripts should be typed on 8 1/2 x 11-inch ordinary white paper. All materials must be double-spaced, including quotations and references. Include a title page on which appears the title of the manuscript, the full name of the author(s) along with position and institutional affiliation, mailing address and telephone number of the contact author. An abstract of 75-150 words should be placed on a separate sheet following the title page. While the title should appear at the top of the first manuscript page, no reference to the author(s) should appear there or any other place in the manuscript. Elements of style, including headings, tables, figures and references should be prepared according to the *Publication Manual of the American Psychological Association, 3rd Edition, 1983*. Figures *must* be camera-ready.

## Submission of Manuscripts

Send four copies of the manuscript to the Editor along with a letter stating that the manuscript is original material that has not been published and is not currently being considered for publication elsewhere. If the manuscript contains copyright materials, the author should note this in the cover letter and indicate when letters of permission will be forwarded to the Editor. Manuscripts and editorial correspondence should be sent to: Richard A. Schwier, *Canadian Journal of Educational Communication, Communications, Continuing & Vocational Education, College of Education, University of Saskatchewan, Saskatoon, Saskatchewan, S7N OWO*.