DOT/FAA/AM-00/12

Michael C. Heil

Brandy O. Agnew

Civil Aeromedical Institute

Federal Aviation Administration

Oklahoma City, Oklahoma 73125

April 2000

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

The Effects of Previous

Computer Experience on Air

Traffic-Selection and Training

(AT-SAT) Test Performance

Office of Aviation Medicine

Washington, D.C. 20591

U.S. Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-00/12

4. Title and Subtitle 5. Report Date

The Effects of Previous Computer Experience on Air Traffic-Selection

and Training (AT-SAT) Test Performance

April 2000

6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Heil, M.C., and Agnew, B.O.

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

11. Contract or Grant No.

FAA Civil Aeromedical Institute

P.O. Box 25082

Oklahoma City, OK 73125

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aviation Medicine

Federal Aviation Administration 14. Sponsoring Agency Code

800 Independence Ave., S.W.

Washington, DC 20591

15. Supplemental Notes

16. Abstract

Many tests that have traditionally been administered in paper-and-pencil format are now administered on a

computer workstation. One disadvantage of computer administered tests, however, is the inadvertent

measurement of extraneous abilities related to prior experience with a computer keyboard or mouse.

Surprisingly, there has been little written in the scientific literature about the use of computers in the selection

process by organizations. The current study examines the relationship between computer experience and test

performance using a computerized selection test. Specifically, some computerized tests, often referred to as

“page-turner” tests, present items that have been adapted to the computer and require the examinee to use the

keyboard to select a response to the questions, which are presented one at a time. Another type of test is that

which takes the form of dynamic virtual scenarios, work samples, or simulations. These may resemble a video

or computer game and often require extensive use of both a mouse and a keyboard. The two main objectives of

this study are: to determine if examinees with more computer experience perform better than examinees with

less computer experience, and to determine if the relationship between computer experience and computerized

test performance remains consistent, regardless of the type of test being presented in the computerized format.

A total of 96 people between the ages of 18 and 30 participated in the study. Computer experience was

measured using the Computer Use and Experience Questionnaire. The personnel selection test used was the

Air Traffic-Selection and Training (AT-SAT) test. The relationship between computer experience and

performance on the computerized selection test was investigated using Pearson’s product-moment correlations

and hierarchical multiple regression. MANOVA and t-tests were also used to identify group differences on the

dependent variables. Education was most predictive of AT-SAT performance, and people with more computer

experience received higher composite AT-SAT scores. Future studies should explore the extent to which

computer experience adds incremental validity over a selection test in predicting job performance, as well as

investigate how training may change the relationship between computer experience and test performance so

that the effect on personnel decision-making is minimized.

17. Key Words 18. Distribution Statement

Air Traffic Control Specialist, Personnel Selection,

Computerized Tests, Computer Experience

Document is available to the public through the

National Technical Information Service,

Springfield, Virginia 22161

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 14

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

1

THE EFFECTS OF PREVIOUS COMPUTER EXPERIENCE ON AIR TRAFFICSELECTION AND TRAINING (AT-SAT) TEST PERFORMANCE

Computers have been used more frequently in test

administration in recent years. Many tests that have

traditionally been administered in paper-and-pencil

format are now administered on a computer workstation (Finegan & Allen, 1994; Lee, 1986; Mead &

Drasgow, 1993). Computers provide a faster and

easier method of assessing performance and cognitive

ability. Additionally, computers provide a means to

measure performance using dynamically changing

stimuli or scenarios (Mead & Drasgow, 1993). One

disadvantage of computer administered tests, however, is the inadvertent measurement of extraneous

abilities related to prior experience with a computer

keyboard or mouse, rather than measurement of

relevant knowledge, skills, abilities, and other characteristics (KSAOs) needed for the job. This issue is

particularly important in the realm of personnel selection, given the legal ramifications and potential adverse

impact that may result from incorporating information

into the selection process that is not job-related. The

purpose of this paper is to explore the relationship

between prior computer experience and performance on

a computerized personnel selection test.

Although the use of computers in the selection

process is becoming increasingly popular, there has

been relatively little written in the scientific literature

about their use by organizations. The effect of computer experience on computerized test performance

has received more attention in the education literature, where researchers have studied two primary

issues related to computerized testing: computer anxiety (Bradley & Russell, 1997; Dimock & Cormier,

1991; Legg & Buhr, 1992; Levine, & DonistaSchmidt, 1998; Powers & O’Neill, 1993) and the

equivalence of paper-and-pencil and computerized

tests (Finegan & Allen, 1994; Mead & Drasgow,

1993). Two findings have generally emerged: 1)

people with more computer experience reported less

computer anxiety, while people with higher levels of

computer anxiety failed to perform as well on computerized tests, and 2) the differences in performance

between paper-and-pencil and computerized tests

were negligible.

The current study examines the relationship between computer experience and test performance

using a computerized selection test. One of the main

objectives of this study was to determine if examinees

with more computer experience perform better than

examinees with less computer experience. More specifically, does previous computer experience give

examinees an advantage when taking a computerized

test? The converse is of particular importance in the

arena of personnel selection: Does a lack of computer

experience put examinees at a disadvantage? In a

study exploring the performance of undergraduate

students on a computerized test of arithmetic reasoning, Lee (1986) found that prior computer experience

improved performance. She observed that computerized testing might discriminate against those who

have not worked with computers. This should be of

great importance to hiring organizations since all

racial groups do not have the same access to computers. According to a survey conducted by the U.S.

Department of Commerce (1999), 46.6% of White

households have a computer, as compared to only

23.2% of Black households and 25.5% of Hispanic

households. Further analyses revealed that white

households have more computers than other racial

groups at all levels of income (U.S. Department of

Commerce, 1999).

Another objective of the current study was to

determine if the relationship between computer experience and computerized test performance remains

consistent, regardless of the type of test being presented in the computerized format. For example

some computerized tests, often referred to as “pageturner” tests, present items that have been adapted to

the computer but could also be presented via another

medium such as paper-and-pencil. In most cases, the

examinee uses the keyboard to select a response to the

questions, which are presented one at a time. Another

type of test is that which takes the form of dynamic

virtual scenarios, work samples, or simulations. The

dynamic, scenario-based items, which may resemble

a video or computer game, often require extensive use

of both a mouse and a keyboard. An examinee with
2

no computer experience may be at a disadvantage

when responding to items that require these computer skills. This proposal would be consistent with

the findings of Young, Broach, & Farmer (1997),

who observed that self-reported video game experience was related to performance on a computer-based

air traffic scenario test.

The literature review and summary of relevant

issues presented above led to the following hypothesis: A relationship exists between computer experience and performance on computerized tests;

specifically, people with prior computer experience

will earn higher scores on the computerized selection

tests. Furthermore, the relationship between computer experience and test performance will be greater

on tests that require the use of a mouse in addition to

the keyboard, such as in the case of dynamic, scenario-based tests.

METHOD

Participants

In an effort to ensure that the participants in this

study were representative of potential applicants,

they were required to meet the basic age and educational qualifications for the air traffic controller job.

A total of 96 participated in the study, 55 male and

41 female, between the ages of 18 and 30: Other

demographic information is presented in Table 1. An

outside personnel agency recruited participants, who

were paid an hourly rate.

Measures

Computer Experience. Computer experience was

measured using the Computer Use and Experience

Questionnaire (CUE) developed by Potosky and

Bobko (1997). This questionnaire consisted of twelve

items (Appendix A) that are answered on a 5-point

Likert Scale (1=Strongly Disagree, 2=Disagree, 3=Neither Agree nor Disagree, 4=Agree, 5=Strongly Agree).

The purpose of administering the questionnaire was

to determine the subject’s knowledge of computers

and to assess the extent to which he or she has used a

computer. These items were summed to create a

composite score. Potosky and Bobko reported a coefficient alpha of .92.

For the purpose of analyses, the composite computer experience score was used as the measure of

computer experience for all analyses other than the

Multiple Analysis of Variance (MANOVA). The

Table 1. Demographics

Variable Number Percent

Age

18-20 years 20 20.8

21-23 years 26 27.1

24-26 years 23 23.9

27-30 years 27 28.1

Gender

Male 55 57.3

Female 41 42.7

Race

Minority 36 39.6

Non-Minority 55 60.4

Education

High School 24 25.3

Trade School 8 8.4

Attend College 49 51.6

College Degree 10 10.5

Graduate School 4 4.2

Table 2. Computer Experience Items

60.4%

I am computer literate.

• Agree

• Disagree 39.6%

57.3%

I regularly use a PC for

word processing.

• Agree

• Disagree 42.7%

52.1%

47.9%

I am good at using

computers.

• Agree

• Disagree
3

item “I regularly use a PC for word processing” was

used individually in the MANOVA to improve understanding of the relationship between the use of a

computer for these specific tasks and performance on

computerized tests. The distribution of responses to

three specific items, “I am computer literate,” “I

regularly use a PC for Word Processing,” and “I am

good at using computers,” is shown in Table 2.

Personnel Selection Test. The personnel selection

test used in the current study is the Air TrafficSelection and Training (AT-SAT) test, a newly developed, computerized test of cognitive ability. This test

will be used by the FAA to select individuals into the

Air Traffic Control Specialist (ATCS) job. The ATSAT battery is comprised of seven tests of cognitive

ability and one non-cognitive measure. The noncognitive measure was included in the AT-SAT composite score but was not used independently in the

current study, as the focus of the study was on

cognitive ability tests. In addition to the AT-SAT

composite score based on total test performance,

scores were also calculated for each cognitive test

included in the battery. A description of each subtest

follows.

The Applied Math test contains 30 multiple-choice

questions. The first five items are practice questions

followed by 25 scored items. An example of an

Applied Math question is: A plane has flown for 3

hours with a ground speed of 210 knots. How far did the

plane travel? Each of these questions requires the

subject to make calculations based on time, speed,

and distance to identify the correct answer from

among four choices.

The Angles test measures the subject’s ability to

recognize the measurement of angles. This test contains 30 multiple-choice questions with four response

options. There are two types of questions on the test.

The first type presents a picture of an angle and

requires the subject to estimate (in degrees) the correct size of the angle. The second type presents a

measure in degrees (e.g. “35o”) and asks the subject to

choose the depicted angle that best represents that

degree measurement.

The Letter Factory test (LF) simulates four factory

assembly lines, each of which manufactures one of

four letters of the alphabet (A, B, C, or D) in one of

three colors. The test requires that subjects use a

mouse to perform multiple and often concurrent

tasks. Each test section begins with letters that appear

at the top of the conveyor belts and move down

toward the loading area. The object of the test is to

“load” each of the colored letters into boxes that

correspond to the letter’s color (e.g., an orange letter

must go into an orange box). Based on the letters on

the conveyor belts, subjects immediately begin selecting and moving boxes to the loading area to

provide just the right number and color of boxes to

correctly place all letters. Other tasks performed

during the simulated factory settings include: (1)

ordering new boxes when supplies become low, and

(2) calling Quality Control when defective letters

appear (i.e., letters that are not As, Bs, Cs, or Ds). The

LF test produces two scores: LF situational awareness

and LF planning and thinking ahead.

The Air Traffic Scenarios Test (ATST) is a lowfidelity simulation of an air traffic control (ATC)

radar screen that is updated every seven seconds. The

goal is to maintain, as efficiently as possible, separation and control of a varying number of simulated

aircraft (represented as data blocks) within the designated airspace. Aircraft in flight can pass through the

airspace or land at one of two airports within the

airspace. Each aircraft’s data block indicates its present

heading, speed, and altitude. There are eight different headings representing 45 degree increments, three

different speeds (slow, moderate, fast), and four different altitude levels (1=lowest and 4=highest). Separation and control are achieved by communicating

and coordinating with each aircraft by using the

computer mouse to click on the data block representing each aircraft and providing instructions such as

changes to the current heading, speed, or altitude.

The ATST produces three scores: AT Efficiency, AT

Safety, and AT Procedural Accuracy.

In the Scan test, subjects monitor a field on the

screen that contains discrete objects (called data

blocks), which are 1⁄2-inch tall and are moving in

different directions. Data blocks appear in the field,

travel in a straight line for a short period of time, and

then disappear. During the test, the subject sees a

blue field that fills the screen, with the exception of

a two-inch white bar at the bottom. In this field, up

to 12 green data blocks may be present. Each data

block contains two lines of letters and numbers

separated by a horizontal line. The upper line is the

identifier and begins with a letter followed by a twodigit number. The lower line contains a three-digit

number. Subjects are scored on the speed with which

they notice and respond to the data blocks that have

a number on the lower line outside a specified range.
4

Throughout the test, this range is displayed at the

bottom of the screen (e.g., 360-710). To “respond”

to a data block, the subject types the two-digit number from the upper line of the block (ignoring the

letter that precedes it), and then presses “enter.”

The Dial Reading test measures the subject’s ability to quickly identify and accurately read certain

dials on an instrument panel. Subjects are asked to

choose from one of five response alternatives for each

question about a given display. The test consists of 20

questions. Individuals pace themselves against the

display of time remaining in the subtest. Subjects are

advised to skip difficult items and return to them at

the end of the test. Each panel consists of seven dials

in two rows, a layout that remains constant throughout the test. Each of the seven dials contains unique

flight information (e.g., air speed, fuel, temperature).

The Analogies test measures the subject’s ability to

apply the correct rules to solve a given problem as well

as their efficiency in using the available information

to solve that problem. Analogies are based on words,

pictures, or figures and appear in three “windows” on

the same screen for a given item. Subjects use a mouse

to move freely between the three windows, view the

different parts of the analogy, and select their answer.

However, they can view only one window at a time.

Window A presents the first part of the analogy,

which requires subjects to infer the underlying rule.

Window B contains the second part of the analogy,

which requires subjects to apply the inferred rule.

Finally, Window C provides subjects the opportunity to confirm their choice by selecting their answer

from the available response options. The test has 57

items: 30 word analogies and 27 visual (i.e., either

pictorial or figural) analogies.

As stated above, one of the objectives of the current study was to determine if performance on certain

types of tests was affected more by computer experience than performance on other types. The ATST

and LF tests are dynamic scenario-based tests that

require the use of the mouse. The scan test is also a

dynamic test but requires use of the keyboard number

pad, rather than a mouse. The analogies test is a static

test that requires use of the mouse to view different

parts of the screen and to select the correct response.

The applied math, angles, and dial reading tests are

static “page-turner” tests that require only the use of

the keyboard to select the correct response for multiple choice questions.

Procedure

 All subjects participated in a pre-screening session

prior to administration of the selection test. During

this time, they were informed of the purpose of the

study, were given voluntary consent forms, and completed biographical information questionnaires. Ten

participants could be tested simultaneously based on

the availability of computer workstations. Participants were randomly assigned to groups of ten. All

computer workstations were separated by partitions.

Participants were given the same instructions before

beginning the test battery. The computer experience

questionnaire was administered at the beginning of

the session as part of the AT-SAT test. Participants

received ten-minute breaks between certain sections

of the test, as well as a 45-minute break for lunch. The

average testing time was approximately six hours.

RESULTS

The relationship between computer experience

and performance on the computerized selection test

was investigated using Pearson’s product-moment

correlations and hierarchical multiple regression.

MANOVA and t-tests were used to identify group

differences on the dependent variables. The results of

these analyses are presented below.

Computer experience/ test score relationship

The AT-SAT subtest correlation matrix, presented

in Table 3, revealed statistically significant correlations between most of the tests within the AT-SAT

battery. Pearson’s product-moment correlations were

used to investigate the linear relationship between

computer experience and AT-SAT scores. As shown

in Table 4, there was a moderate correlation between

computer experience and AT-SAT composite score

(r=.38, p<.01). An examination of relationships between computer experience and individual AT-SAT

subtest scores revealed that computer experience was

not significantly correlated with the dial reading,

scan, and AT-Scenarios procedural accuracy scores.

Computer experience was most highly correlated

with the letter factory (LF) situational awareness

(r=.45, p<.01), applied math (r=.37, p<.01), and AT

efficiency (r=.365, p<.01) measures. As demonstrated

by these correlations, and contrary to the hypothesis,
5

Table 3. Correlation of AT-SAT Sub-tests

1 2 3 4 5 6 7 8 9 10 11 12

1. Dials 1.00

2. Angles .603* 1.00

3. Amath .559* .702* 1.00

4. LF SA .430* .430* .599* 1.00

5. LF TP .428* .527* .560* .603* 1.00

6. Analogies .617* .753* .705* .576* .511* 1.00

7. Scan .377* .410* .496* .422* .550* .465* 1.00

8. AT Eff. .514* .479* .625* .691* .549* .522* .422* 1.00

9. AT Safe .146 .107 .198 .415* .169 .143 .005 .368* 1.00

10. AT PA .413* .367* .379* .284* .340* .353* .375* .440* -.119 1.00

11. AT Total .490* .434* .562* .681* .495* .471* .353* .868* .686* .518* 1.00

12. AT-SAT

Composite

.727* .846* .913* .691* .699* .835* .633* .715* .246 .477* .671* 1.00

* Correlation is significant at the 0.01 level (2-tailed)

Table 4. Correlation of AT-SAT Subtests

With Computer Experience

Test Computer Experience

Score

Dials .196

Angles .266**

Amath .372**

LF SA .447**

LF TP .283**

Analogies .339**

Scan .199

AT Eff. .365**

AT Safe .207*

AT PA .054

AT Total .318**

AT-SAT

Composite

.380**

** Correlation is significant at the 0.01 level (2-tailed)

* Correlation is significant at the 0.05 level (2-tailed)
6

no clear pattern of relationship emerged between

computer experience and test performance based on

whether the test was static or dynamic.

Hierarchical regression was used to determine the

amount of variance in test performance that was

explained by computer experience after age and education were taken into account. The AT-SAT composite test score and each AT-SAT subtest score were

used as dependent variables in a series of separate

regression analyses. As shown in Table 5, age did not

contribute significantly to the prediction of the ATSAT composite score. When entered into the regression equation, education contributed significantly to

the prediction of AT-SAT score (∆R2

=.217, p<.01).

Computer experience, which was then entered into

the equation after education, resulted in a significant

R2

 change in predicting the AT-SAT score (∆R2=.039,

p<.05). Based on the adjusted R2, the overall model of

age, education, and computer experience accounted for

23.3% of the variance in the composite AT-SAT score.

A summary of the regression analyses for each ATSAT subtest is provided in Table 6. For each analysis,

the independent variables were entered in the following order: age, education, and computer experience.

A separate analysis was performed for each AT-SAT

subtest score. Age did not contribute significantly to

the prediction of any of the subtest scores. Education

added significantly to the prediction of the following

scores: Analogies (∆R2

=.242, p<.01), Applied Math

(∆R2

=.218, p<.01), Angles (∆R2=.179, p<.01), LF

Situational Awareness (∆R2

=.148, p<.01), Dial Reading (∆R2

=.094, p<.01), AT Total (∆R2=.091, p<.01),

AT Efficiency (∆R2=.088, p<.01), AT Safety

(∆R2

=.083, p<.01), and LF Planning and Thinking

Ahead (∆R2

=.055, p<.01). Once age and education

were entered into the regression equation, computer

experience added significantly only to the prediction

of LF Situational Awareness (∆R2=.091, p<.01), AT

Efficiency (∆R2

=.064, p<.05), LF Planning and

Thinking Ahead (∆R2

=.048, p<.01), and AT Total

(∆R2

=.041, p<.05). The results of the regression

analysis support the hypothesized relationship between computer experience and performance on dynamic tests requiring use of a mouse. The regression

weights and adjusted R2

 for each separate analysis are

listed in Table 6.

Group comparisons

Differences in mean computer experience scores

based on gender, race, and education were investigated using t-tests. The results, summarized in Table

7, revealed no significant gender or racial differences

in mean computer experience score. However, participants who either attended or graduated from

college had a significantly higher computer experience score than those with a high school or trade

school education, t=-3.28 (92), p<.05.

A 2X2 between subjects MANOVA was performed

on the ten AT-SAT subtests. Education and selfreported word-processing experience served as the

independent variables. Order of entry of the independent variables was education, then word processing

experience. Using Wilks’ criterion, there was a significant main effect of both education, F(10,

82)=2.75, p<.01, and word processing experience,

F(10, 82)=2.14, p<.05. The interaction was not significant. The results reflected a moderate association

between both education (η2

=.25) and word processing experience (η2

=.21) and the combined dependent

variables (DVs). Differences in AT-SAT test scores

associated with age and education are presented in

Appendix B.

Table 5. Hierarchical Multiple Regression of Age, Education, and Computer Experience

on Composite AT-SAT Score

ATSAT

Comp.

Age Educ. B β ∆R2

Age -.045 -.196 -.048 .002

Education .456** .129 4.642 .375 .217**

Computer

Experience

.380** .205* .389** .317 .224 .039*

R2=.258*

Adjusted R2

= .233

R=.508

*p<.05

**p<.01
7

Table 6. Regression of Age, Education, and Computer Experience on

AT-SAT Sub-tests

Dependent

Variable

Independent

Variables

β ∆R2 Adjusted R2

Dials Age -.096 .005

Education .280 .094**

Computer Experience .068 .004

.073

Angles Age -.070 .002

Education .382 .179**

Computer Experience .103 .008

.162

Amath Age .019 .000

Education .374 .218**

Computer Experience .231 .042

.236

LF SA Age -.053 .008

Education .244 .148**

Computer Experience .341 .091**

.223

LF TP Age .076 .002

Education .131 .055*

Computer Experience .247 .048*

.074

Analogies Age -.091 .004

Education .432 .242**

Computer Experience .152 .018

.240

Scan Age -.045 .006

Education .028 .011

Computer Experience .179 .025

.010

AT Eff. Age -.050 .007

Education .178 .088**

Computer Experience .285 .064*

.131

AT Safe Age -.153 .018

Education .259 .083**

Computer Experience .075 .004

.076

AT PA Age .184 .023

Education -.067 .000

Computer Experience .117 .011

.002

AT Total Age -.042 .004

Education .208 .091**

Computer Experience .228 .041*

.107

*p<.05

**p<.01
8

A stepdown analysis was performed on the prioritized DVs to investigate the effect of each main effect

on the individual DVs. The DVs were prioritized

according to their incremental validity in predicting

job performance (Ramos, 1999). In stepdown analysis, each DV was analyzed with higher priority DVs

treated as covariates and tested as a univariate ANOVA

(Tabachnick & Fidell, 1996). Results of the stepdown

analysis are presented in Table 8. Applied math

provided a unique contribution to predicting differences between those with high school/trade school vs.

college education, stepdown F(1, 91)=12.75, p<.01.

Participants who attended college scored higher on

the applied math test (M=12.35) than did those with

a high school or trade school education (M=8.77).

After the pattern of differences measured by applied

math was entered, a difference was also found on the

analogies test, stepdown F(1, 90)=4.70, p<.05. The

participants who attended college scored higher on

the analogies test (M=4.14) than did those participants with a high school or trade school education

(M=3.08). Although a univariate comparison revealed that those with more education scored higher

on the angles test, univariate F(1, 91)=9.10, p<.01,

LF situational awareness, univariate F(1, 91)=4.58,

p<.05, and the dial reading test, univariate F(1,

91)=4.58, p<.05, these differences were already represented in the stepdown analysis by higher-priority DVs.

Table 7. Group Means and Results of t-Test

of Computer Experience

Group Computer

Experience Score

Mean (SD)

Gender

• Male 39.75 (9.55)

• Female 37.20 (9.77)

Race

• Minority 38.03 (8.15)

• Non-Minority 39.00 (10.88)

Education

• High School 34.19 (7.13)

• College 40.83 (10.07)*

*p<.05

As shown in Table 8, applied math made a unique

contribution to predicting differences between those

with and those without word processing experience,

stepdown F(1, 91)=4.44, p<.05. Participants with

experience using a computer for word processing

scored higher on the applied math test (M=11.62)

than did those with no word processing experience

(M=9.49). With differences due to the applied math

score already entered, the analogies test score contributed uniquely, stepdown F(1, 90)=8.14, p<.01. Participants with experience using a computer for word

processing scored higher on the analogies test

(M=4.09) than did those with no word processing

experience (M=3.12). Although a univariate comparison revealed that participants with word processing experience scored higher on LF situational

awareness, univariate F(1, 91)=11.01, p<.01, LF

thinking and planning ahead, univariate F(1,

91)=6.76, p<.05, and AT efficiency, univariate F(1,

91)=7.49, p<.01, these differences were already represented in the stepdown analysis by applied math

and analogies scores.

DISCUSSION

The results of the analyses described above provide

evidence of a relationship between prior computer

experience and performance on a computerized personnel selection test. In general, these results are

consistent with those reported by Keenan (1999)

during the AT-SAT validation study. The results of

the Pearson product-moment correlations reported

in the current study provide clear evidence of a

positive relationship between computer experience

and the AT-SAT composite score. Examination of

the correlations between computer experience and

the individual AT-SAT subtest scores provides insight into which of the tests are more affected.

Only three of the subtest scores (dial reading, scan,

and AT procedural accuracy) were not correlated

with computer experience. The lack of a relationship

between dial reading score and computer experience

is not surprising given that the dial reading test is a

static “page-turner” multiple choice test. The examinee need only use the keyboard to select a particular

response item. Since the scan and AT procedural

accuracy scores are based on dynamic tests, the lack of

a relationship between performance and computer

experience is more surprising. However, closer inspection of the manner in which these scores are
9

Table 8. MANOVA of Education, Word Processing Experience, and Their Interaction

IV DV Univariate

F

Df Step-down

F

Df

Amath 12.57** 1/91 12.57** 1/91

Analogies 15.70** 1/91 4.70* 1/90

Angles 9.10** 1/91 .03 1/89

LF SA 4.58* 1/91 .01 1/88

LF TP 1.39 1/91 .98 1/87

AT Safe 2.73 1/91 1.88 1/86

AT Eff. 2.88 1/91 .40 1/85

AT PA .09 1/91 3.30 1/84

Dials 4.58* 1/91 .00 1/83

Education

Scan .05 1/91 2.71 1/82

Amath 4.44* 1/91 4.44* 1/91

Analogies 13.08** 1/91 8.14** 1/90

Angles 3.78 1/91 .57 1/89

LF SA 11.01** 1/91 3.52 1/88

LF TP 6.76* 1/91 .45 1/87

AT Safe .92 1/91 .06 1/86

AT Eff. 7.49** 1/91 .31 1/85

AT PA .26 1/91 2.08 1/84

Dials 1.02 1/91 1.28 1/83

Word Processing Experience

Scan 2.89 1/91 .22 1/82

Amath 7.00* 1/91 7.00* 1/91

Analogies 3.33 1/91 .11 1/90

Angles .98 1/91 1.30 1/89

LF SA 4.27* 1/91 .51 1/88

LF TP 1.28 1/91 .11 1/87

AT Safe .65 1/91 .08 1/86

AT Eff. 2.20 1/91 .11 1/85

AT PA .03 1/91 .91 1/84

Dials 2.29 1/91 .29 1/83

Education and Word

Processing

Scan .04 1/91 3.38 1/82

*p<.05

**p<.01
10

derived provides some insight into this issue. The

scan test requires the use of the keyboard for input of

responses; the mouse is not used for this test. The AT

procedural accuracy score is based on whether or not

the examinee follows the correct procedures when

directing the traffic in each scenario. For example, all

planes must land at the airport, at the slowest speed

and the lowest altitude. These rules remain the same

regardless of the scenario. Quick and efficient mouse

movement is not required for an examinee to score

well on AT procedural accuracy.

Contrary to what was hypothesized, performance

on the applied math and angles tests (which are static

multiple choice tests) was correlated with computer

experience. Further investigation revealed that this

relationship might be due to the educational level of

the examinee. The results of t-tests indicate that

people with more education also had more computer

experience. Hierarchical multiple regression revealed

that computer experience did not add anything beyond education in predicting applied math or angles

scores. Consequently, it was concluded that the significant correlation between computer experience

and performance on these static tests was due to

examinee education level.

Computer experience added significantly to the

prediction of the dynamic, scenario-based tests that

require the use of a mouse. After controlling for

education, computer experience added to the prediction of LF situational awareness score, LF planning

and thinking ahead score, and AT efficiency score.

These results demonstrate that, after controlling for

education, computer experience was related only to

performance on dynamic scenario-based tests that

require use of a mouse. The LF and ATST items

require fast and efficient mouse manipulation. The

relationship between computer experience and performance on these subtests was enough to influence

the relationship between computer experience and

performance on the overall AT-SAT composite score.

The hierarchical multiple regression with AT-SAT

composite as the DV revealed that computer experience produced a significant change in R2

 beyond

education. Consequently, people with more computer experience received higher composite AT-SAT

scores. Whether or not this relationship is due to

overall experience with a computer or to experience

with a mouse in particular is not known; the computer experience questionnaire did not contain a

question specific to the use of a mouse. Such a

question should be included in the future as it has

implications for training interventions.

The results of the MANOVA stepdown analysis

suggest that the pattern of differences on AT-SAT

subtests between people with high school/ trade school

education and college education were best measured

by scores on applied math and analogies. The same is

true of the differences between people with computer

word processing experience vs. those without computer word processing experience. Consequently, the

influence of the tests that provide an advantage to

those with computer experience (LF, ATST) is minimal when the subtest scores are combined in a linear

fashion based on their incremental validity in predicting job performance.

In summary, the results revealed that people with

a higher level of education also had more computer

experience. Once education level was controlled for,

computer experience was correlated with performance

on dynamic tests that required the use of a mouse.

This relationship was sufficient to influence the selection test composite score, as demonstrated by the

finding that computer experience added to prediction of the AT-SAT composite score even after controlling for education. The influence of these dynamic

tests on the composite score might be a function of

the weighting of these tests in the composite’s calculation. Education was found to be most predictive of

performance on AT-SAT. Currently, any adverse

impact associated with the AT-SAT battery is unknown since the test is not yet operational. However,

people with less computer experience may be at a

disadvantage when taking a computerized test that

requires the use of a mouse to complete a dynamic

test. Since people of all races do not have equal access

to computers (U.S. Department of Commerce, 1999),

this disadvantage may contribute to adverse impact.

The potential for disadvantage will depend on the

number of such tests included in a selection battery,

as well as their weight in calculating the composite

score.

The relationship between computer experience

and performance on a computerized selection test has

important implications for personnel decision making. If dynamic scenario-based tests requiring the use

of a mouse are heavily weighted in the calculation of

a selection test score, the ranking of examinees among

qualified applicants may be affected by whether or
11

not they had previously used a computer. Future

studies should explore this issue as well as the extent

to which computer experience adds incremental validity over a selection test in predicting job performance. Items pertaining to mouse and video game

experience should also be added to the computer

experience questionnaire, as these may help isolate

the specific types of training that are most likely to be

effective in reducing disparity between those with

and without prior computer experience. This can

assist in the development of a computer training

intervention. It may be that such training need only

focus on mouse usage, rather than on such things as

operating systems and databases. Future research

should also investigate the extent to which such

training may change the relationship between computer experience and test performance so that the

impact on personnel decision-making is minimized.

REFERENCES

Bradley, G., & Russel, G. (1997). Computer experience, school support, and computer anxieties.

Educational Psychology, 17, 267-84.

Dimock, P.H., & Cormier, P. (1991). The effects of

format differences and computer experience on

performance and anxiety on a computer administered test. Measurement and Evaluation in Counseling and Development, 24, 19-126.

Finegan, J.E., & Allen, N.J. (1994). Computerized and

written questionnaires: Are they equivalent? Computers in Human Behavior, 10, 483-96.

Keenan, P.A. (1999). Biographical and computer experience information: Demographics for the validation study. In Ramos, R., Heil, M.C., & Manning, C.A. (Eds.). Documentation of Validity for

the AT-SAT Computerized Test Battery. (DOT/

FAA/OAM Technical Report). Manuscript under review.

Lee, J.A. (1986). The effects of past computer experience on computerized aptitude test performance.

Educational and Psychological Measurement, 46,

727-33.

Legg, S.M., & Buhr, D.C. (1992). Computerized adaptive testing with different groups. Educational

Research: Issues and Practice, 11, 23-27.

Levine, T., & Donitsa-Schmidt, S. (1998). Computer use, confidence, attitudes, and knowledge: A causal analysis. Computers in Human

Behavior, 14, 125-46.

Powers, D.E., & O’Neil, K. (1993). Inexperienced and

anxious computer users: Coping with a computer

administered test of academic skills. Educational

Assessment, 2, 153-73.

Potosky, D., & Bobko, P. (1997). Assessing computer

experience: The computer understanding and experience scale (CUE). Poster presented at the Society

for Industrial and Organizational Psychology

(SIOP), April 12, St. Louis, MO.

Ramos, R. (1999). AT-SAT: Overview of an ambitious

project. Presented at The Society for Industrial

and Organizational Psychology’s 1999 Annual

Conference, Atlanta, GA.

Tabachnick, B.G., & Fidell, L.S. (1996). Using Multivariate Statistics (3rd Ed.). New York; Harper Collins College Publishers.

U.S. Department of Commerce. (1999). Falling through

the net: Defining the digital divide. Washington,

D.C.: National Telecommunications and Information Administration (NTIS No. PB99-1444

87/INZ).

Young, W.C., Broach, D., & Farmer, W.L. (1997).

The effects of video game experience on computerbased Air Traffic Controller Specialist, Air Traffic

Scenario test scores. (DOT/FAA/AM-97/4). Washington, DC: Federal Aviation Administration

Office of Aviation Medicine. Available from:

National Technical Information Service, Springfield, VA 22161; order # ADA322774.
12
A1

APPENDIX A

Computer Use and Experience Questionnaire

1. I frequently read computer magazines or other sources of information that describe new computer

technology.

2. I know how to recover deleted or “lost data” on a computer or PC.

3. I know what LAN is.

4. I know what an operating system is.

5. I know how to write computer programs.

6. I know how to install software on a personal computer.

7. I know what e-mail is.

8. I know what a database is.

9. I am computer literate.

10. I regularly use a PC for word processing.

11. I often use a mainframe computer system.

12. I am good at using computers.
A2
Age Education

AT-SAT Score 18-21 22-26 27-30 F H.S. or Trade College F

Amath 12.15

(5.03)

12.09

(5.31)

10.87

(5.45)

.54 8.82

(4.61)

13.14

(4.93)

17.0

*

Analogies 4.07

(1.26)

4.01

(1.51)

3.68

(1.56)

.59 3.04

(1.28)

4.38

(1.33)

22.2

*

Angles 11.65

(3.33)

11.20

(3.51)

10.83

(3.38)

.40 9.53

(3.70)

12.05

(2.92)

13.2

*

LF SA 1.87

(.81)

1.72

(.76)

1.52

(.71)

1.5 1.39

(.61)

1.87

(.79)

8.8*

LF TP 3.71

(1.15)

3.75

(1.11)

3.65

(1.01)

.06 3.41

(1.04)

3.86

(.18)

3.7*

AT Safe 2.89

(.89)

2.96

(.80)

2.58

(.70)

1.9 2.60

(.92)

2.95

(.73)

4.1*

AT Eff. 2.70

(.75)

2.72

(.88)

2.39

(.55)

1.8 2.35

(.68)

2.75

(.78)

6.0*

AT PA 1.85

(.59)

1.87

(.51)

2.05

(.49)

1.3 1.93

(.60)

1.91

(.50)

.03

Dials 9.92

(1.36)

9.91

(1.69)

9.48

(2.57)

.51 9.10

(2.08)

10.13

(1.71)

6.6*

Scan 8.10

(1.69)

8.45

(1.99)

8.31

(1.79)

1.1 8.40

(1.80)

8.67

(1.88)

.45

*p<.05

Standard Deviation in ( ).

APPENDIX B

Mean AT-SAT Scores by Age and Education

B-1
B2