COMPUTER SIMULATION OF HUMAN RATINGS OF THE PARAGRAPH COMPLETION METHOD

Summ5ry.-A computer-based algorithm for simulating human ratings of the Paragraph Completion Method, an open-ended response technique for assessing conceptual level, was assessed. The results indicate that a reasonably reliable, efficient approximation to human ratings may be achieved through the use of a computer. Advantages to computer-based simulation of ratings are reduced costs of training human raters and reduced fluctuations in reliability. The use of open-ended response instruments in psychological research has historically been linked to the practical problem of training multiple raters in coding skills. Such training is notoriously time consuming and the resulting interrater reliability coefficients are variable. Further, raters' stability and interrater agreement deteriorate over several protocols, so retraining is typically necessary to maintain reliability on large scale projects (1). The availability of an automated, objective and reliable method of scoring open-ended protocols would reduce the risk of fluctuating inter-rater reliability and would facilitate the use of open-ended response instruments in psychological research. As a case in point, Hunt (3, 4) has advocated the use of the Paragrzph Completion method as an effective instrument for measuring conceptual level, a unique organizing property of personality which includes conceptual complexity and is developmentally related. Hunt further states that conceptual level is related to differential achievement under alternative environmental constraints. The Paragraph Completion method is a semiprojective, written technique using six open-ended topical stems to assess the individual's concepts and attitudes regarding conflict, uncertainty, rules, and authority. Protocols are rated to reflect level of conceptual complexity and socialization. Responses of low conceptual level reflect rigid, concrete thinking while responses of a higher level reflect multifaceted, flexible thinking. As with similar instruments, considerable training and practice is needed to bring raters to acceptable levels of reliable coding. In research reviewed by Hunt, et al. (5) interrater agreements for the composite rating have ranged from .75 to .74. Given that the Paragraph Completion method is said to measure conceptual level which reflects general cognitive functioning, responses should also covary on language ad syntactic complexity. Since considcrable effort has already been directed at computer-based analysis of textual materials, some of the heuristic principles generated in those studies should generalize to the computer-based assessment of open-ended response formars such as the Paragraph Completion method. The purpose of this study was to develop a computer-based algorithm for identifying lexical features of protocols and to assess the usefulness of those features in simulating human ratings of the protocols. Sample protocols (n = 162) from the manual (5) for training raters were entered verbatim into computer textual files. The files were then subjected to analysis by a text analytic program. The program, PCMANAL, is written in Pascal language and generates a series of measures on each protocol. Initially, the program counts the number of total words (Tokens), total different words (Types), words not in a dic