UNITED STATES AIR FORCE RESEARCH LABORATORY PILOT SELECTION METHODS

ions such as "airmanship," "intelligence," "situational awareness," or "workload" are called constructs. They cannot be observed directly and must be inferred from some test, measurement scale, or questionnaire that operationalizes the components of the construct. There is no scientific value to a construct that cannot be measured. It is important to remember that a construct can be measured by many means. Hunter and Hunter (1984) have demonstrated this and Spearman (1927) noted it in his idea of "the indifference of the indicator" as Jensen (1993) has pointed out. Walters, Miller, and Ree (1993) have labeled the specious reasoning that because two tests look different they must measure different constructs, the "topographical fallacy. " The appearance of the items or tasks of a test is not a reliable indicator of what is being measured. Results of a construct validation study provide a good indicator of what is being measured. Consider the following example. Several NATO countries use interviews as part of their military pilot selection process (Hansen, 1999). On the basis of the topographical fallacy (i.e., differing appearances), the U. S. Air Force (USAF) considered using a structured interview in their pilot selection process (Walters et al., 1993). Structured interviews have predetermined rules for obtaining, observing, and evaluating responses. In a recent review of the literature on employment interviews, Whetzel and McDaniel (1997) concluded that structure tends to make the interview more reliable and valid. In their meta-analytic review of the utility of personnel selection methods, Schmidt and Hunter (1998) concluded that structured interviews were incrementally valid to measures of general cognitive ability for predicting training and job performance. Walters et al. (1993) reported that the highly structured USAF pilot selection interview measured educational background, motivation to fly, self-confidence and leadership, and flying job knowledge. Additionally, three broad ratings of probable success in pilot training, bomber-fighter flying, and tanker-transport flying were made. The sample was 223 USAF pilot trainees who were administered the structured interview, Air Force Officer Qualifying Test (AFOQT; Skinner & Ree, 1987), and computer based cognitive "information processing" (verbal classification, mental rotation, and short-term memory) and personality (self-confidence and attitudes toward risk) tests. Passing-failing pilot training was the criterion. The seven structured interview scores had an average validity of . 21. When the seven scores were added to regression equations containing the AFOQT scores (subtest average validity of .28) and the computer-based test scores (average validity of .18), no incremental validity was found. Despite the difference in appearance between the paper-and-pencil AFOQT and the interview, the interview was not able to account for any unique prediction of pilot performance. It is suggested that researchers and practitioners in pilot selection administer their selection instruments along with a battery of tests of known constructs to have a better understanding of what is being measured by their selection instruments. Misinterpretation of Factor Analytic Results Factor analysis (Spearman, 1904, 1927) is the name given to a group of statistical techniques for determining the latent or unobservable sources of variance in a correlation matrix. In personnel selection, it is commonly used to identify the constructs measured by a test battery or by job performance criteria. Rotation of the initial factor solution plays an integral role in factor analysis and can produce a problem in interpretation. During rotation the variance from the first factor is spread across all the factors. This can make the first factor seem to disappear when in reality it has simply been distributed across all the factors. The solution is to either not rotate or use a residualized (Schmid & Leiman, 1957) hierarchical solution. Lack of Statistical Power Statistical power is the probability of detecting a significant effect, such as a difference between means or a non-zero correlation, when present. Specifically, it is the probability of rejecting a false null hypothesis (Cohen, 1987). Although the topic is covered in most introductory statistics classes, relatively few published studies report power for the test statistics (such as F, t, or z,) used. See for example Ree and Earles (1991, 1993) and Walters et al. (1993). Sedlmeier and Gigerenzer (1989) reported two surveys of a prestigious applied psychology journal and showed that the average statistical power for published studies was only . 46 and fell to . 37 two decades later. This means that the researchers could only detect an existing effect 46 percent (or 37 percent) of the time. On the other hand, 54 percent of the time (or 63 percent of the time) they would fail to find the existing effect! No researcher should conduct a study with less than a high chance of detecting a significant result. Low statistical power makes it more likely that we will draw incorrect conclusions from our studies. Too many pilot selection studies have been performed with small samples that inevitably yield low statistical power. Cohen (1987), noted that statistical power is a joint function of the degree to which the sample values reflect their true values in the population (i.e., the reliability of the sample values), effect size, Type I error rate (significance level), and sample size. Before conducting a study, power tables (Cohen, 1987) should be consulted. An informative discussion of sample size requirements (see especially page 222) is given by Schmidt and Hunter (1978). Schmidt, Hunter, and Urry (1976) provide tables showing sample sizes necessary for sufficient statistical power in validation studies given varying selection ratios, reliabilities, and effect sizes A single rule-ofthumb cannot be given. In general, the higher the selection ratio and the less reliable the variables are, the larger the sample must be. Failure to Cross-Validate Cross-validation refers to the use of a regression equation computed in one sample being applied in another sample. This is done to determine the extent to which the initial regression solution can be expected to generalize to another sample. The stability and generalizability of the regression solution are important because the results of the regression equation will be used to make personnel selection decisions about people who were not in the original validation sample. In general, the correlation of the several predictors with the criterion will go down in this second or cross-validation sample (Wherry, 1975). This reduction of the correlation is called shrinkage from overfitting. The classic paradigm for cross-validation was provided by Mosier (1951) in which a single sample is drawn from a population and then divided into separate validation and crossvalidation samples. Murphy (1983) has pointed out that the correlation in the cross-validation sample is still a consequence of overfitting, as there was only one sampling from the population. Moreover, even if there were two samplings from the population, the validation and crossvalidation multiple correlations would be only two values out of a virtually infinitely large set of values. That is why we recommend the use of Stein's operator (Stein, 1960). Kennedy (1988) demonstrated the accuracy of Stein's formula for estimating the mean of the distribution of all possible cross-validated correlations from the population from which the sample was selected. Stein's operator has the advantage of allowing estimates on the largest available sample while offering an estimate of cross-validity. Misinterpretation of Correlations and Regression Selection studies use correlation and regression as a general model and analytic technique. Predictors of success in pilot job performance are correlated with measures of success in pilot job performance. There is an extensive literature on correlation and regression (Cronbach, 1971; Messick, 1989). In the next sections we discuss several issues that can lead to the misinterpretation of correlations and regression, and provide solutions. Holding job experience constant. Ability research is generally correlational and the interpretation of correlations can be fraught with hazards. As an example, consider the correlation of an ability test and ratings of pilot job performance. "Artificially" low correlations that could lead to inappropriately abandoning the ability test can occur for a variety of reasons including the effects of range restriction, unreliability, and the influence of moderating variables. Range restriction and unreliability will be discussed later in this section. It is noted in the Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 1987) that the relationship between ability (or any other measure) and occupational criteria is best represented with the effect of job experience removed. This can be done easily by using partial correlation and "partialing-out" experience from the relationship between ability and the occupational criteria. Carretta, Perry, and Ree (1996) provided an example. When they correlated ability test scores with ratings of situational awareness (SA) for 171 F-15 pilots, the observed correlation of ability and SA was . 10. However, when F-15 flying experience was partialed-out, the correlation was . 17, an increase of 70% in predictive efficiency. It would have been incorrect to report the correlation of ability and SA as . 10. The idea of partial correlation can be subsumed under "mediation," which means that one variable acts through another variable to exert its influence on a third variable. For instance, "A —>B-> C" indicates that variable A acts through variable B to exert its influence on variable C. In this instance there is no direct influence of A on C and we do no