Using Neural Network and Traditional Psychometric Procedures in the Analysis of Test Scores: An Exploratory Study

The major purpose of this exploratory study was to investigate the feasibility of using an artificial neural network as an alternative approach parallel to that of the employment of three traditional psychometric procedures in the analysis of test scores. The database was derived from scores on six 15-item factor scales of the Dimensions of Self-Concept (DOSC), Form W prepared by Crowder and Michael (1991) and administered to a sample of 119 individuals who were employed in an electronics/computer firm. It was concluded that use of neural network methodology as a pattern recognition approach appears to show promise as a potentially viable alternative in test development to the use of traditional psychometric procedures in the preliminary demonstration of internal consistency of scores on factor scales, the intercorrelations of scores on such scales in the context of providing evidence of discriminant validity, and the discrimination power of items within a factor scale. The use of artificial neural networks has become an area of considerable interest in the study of cognitive processes in learning and decision making (Gallant, 1993) and in pattern recognition (Nigrin, 1993). To the knowledge of the writers of this paper, relatively little, if any, work has been done in the exploration of the possible use of artificial neural networks in the analysis of test scores associated with test development. The major purpose of this exploratory study was to investigate the feasibility of using an artificial neural network as an alternative approach parallel to that of the employment of three traditional psychometric procedures in the analysis of test scores. The database was derived from scores on six 15-item factor scales of the Dimensions of Self-Concept (DOSC), Form W prepared by Crowder and Michael (1991) and administered to a sample of 119 individuals who were employed in an electronics/computer firm. In this paper, attention is directed toward and delimited to showing an analogy in the outcomes obtained from (a) the use of each of three selected conventional procedures in psychometric methodology and (b) parallel application of artificial neural network methodology corresponding to each of the three traditional psychometric procedures. Specifically, information was sought concerning the degree of comparability of data resulting from the application of classical test theory and the use of artificial neural network methodology in obtaining indicators or indexes reflecting (a) internal-consistency of scores in each of the six 15-item factor scales of the Dimensions of Self-Concept (DOSC), Form W measure, (b) degree of association or extent of interrelationships of scores among the six factor scales that were anticipated to be relatively low in light of the initial work carried out by Crowder and Michael, and (c) power of item discrimination of contrasting sets of five items judged by the investigators and an external psychometrician on the basis of available data to be relatively high or relatively low within each of the six factor scales. If comparability could be found in the results furnished by neural network methodology to those realized from application of three selected traditional psychometric procedures, use of neural networks might be a plausible alternative in the preliminary stages of test development. One focus, or important objective, of this investigation was that of the realization of evidence of discriminant validity for a multidimensional test such as the DOSC, Form W. From a correlation matrix of scores on 18 subtests (three five-item subtests for each of the six factor scales), Crowder and Michael endeavored to achieve discriminant validity in terms of (a) finding a rotated factor structure in which the three subtests comprising each factor scale tended to show high loadings (weights) typically on one factor dimension and relatively low loadings on all other dimensions (simple factor structure) and (b) in light of the attainment of this simple factor structure relatively low intercorrelations of scores on the six factor scales. …