PART III. TECHNICAL INFORMATION

Items:

a.  Abstracts of Thesis

The objective of this project to empirically validate the job-related
Air Traffic Color Vision (ATCOV) tests for occupational screening of air
traffic controllers. In the ATCOV validation grant, University of
Oklahoma (OU) collected and analyzed data from both color vision
deficient and color normal subjects in two phases. During the phase one,
the ATCOV tests were initially validated, and modifications were made to
improve the test. Data from 82 color vision deficient and 26 color
normal subjects were collected. During the phase two, the modified
version of ATCOV were further validated. Data from 73 color vision
deficient and 151 color normal subjects were collected. The sensitivity,
specificity, and reliability of the ATCOV tests were calculated based on
the collected data. The data analysis results show that the ATCOV tests
are reliable with respect to internal consistency, repeatability,
consistency between parallel versions, and testing monitors (LCD vs.
CRT). The modified ATCOV tests have high sensitivity and specificity.
With the current cut-off scores, ATCOV produce a specificity of 0.97 and
sensitivity of 0.98. Such results support the deployment of ATCOV tests
in the job selection of air traffic controllers.

c.  Data on Scientific Collaborators:

Research Assistant, Chintan Barbhaya, graduate student, School of
Industrial Engineering

e.  Technical Description of Project and Results (see next page for
    final report draft)

**VALIDATING FAA'S JOB-BASED COLOR VISION TESTS FOR AIR TRAFFIC
CONTROLLER APPLICANTS**

**FINAL REPORT**

**Chen Ling, PI**

**UNIVERSITY OF OKLAHOMA**

**PREPARED FOR FEDERAL AVIATION ADMINISTRATION (FAA)**

**CIVIL AEROSPACE MEDICAL INSTITUTE**

**Dr. Jing Xing**

**June, 2008**

**Introduction**

This report summarizes the research conducted in the Cognitive Analysis
and System Engineering (CASE) lab at OU, as part of the project
"Validating FAA's Job-based Color Vision Tests for Air Traffic
Controller Applicants", sponsored by the FAA's Civil Aerospace Medical
Institute (CAMI). The research was carried out over the period spanning
from July 14, 2007 to June 11, 2008.

The job-related Air Traffic Color Vision (ATCOV) tests were developed by
researchers in FAA CAMI to mimic basic air traffic control display
elements and fundamental color use tasks in air traffic control. The
purpose of the test is to examine the applicant's ability to use colors.
The ATCOV consists of four separate tests, each with a unique purpose.
Test 1 examines the color naming ability of the applicant. Subjects are
asked to click on the datablocks whose central texts are of the
specified color names. Ten basic colors were used in the test 1. Test 2
examines the ability of the applicant to use colors in multi-task
scenario. Subjects are presented with a datablock whose central text are
of a particular color, and are asked to click on the datablocks with the
same central text color after completing a math problem on a mask page.
Presenting a math problem on the mask page mimics the multi-task
scenario that the controllers face. Another ten colors were used in test
2. Test 3 examines the applicant's ability to detect a red target among
many other colors. Such ability is important for controllers because
many critical alert messages are presented in red in the air traffic
control system. Subjects are asked to indicate whether a red datablock
appears on the display, and if so, whether it is to the right or left
side of the display. Test 4 examines the ability of the applicant to
read colored text. Subjects are presented with nine datablocks in
multiple colors arranged in a three-by-three matrix, and are asked to
read the datablock contents and click on the datablock that has exactly
the same call sign as the one in the middle. These four tests aim to
help select job applicants with specific color use abilities necessary
to perform air traffic control tasks.

The purpose of the validation grant is to empirically validate the ATCOV
tests for occupational screening by calculating the sensitivity,
specificity, and reliability of the tests. In the ATCOV validation
grant, University of Oklahoma (OU) collected and analyzed data from both
color vision deficient (CVD) and color normal (CN) subjects. Because the
Dvorine color vision test has been used for the FAA's initial screening
of the controllers, and is also a widely-used and well-validated
clinical test, we chose the Dvorine color vision test as the golden
standard for this validation study. Reliability of the test consists of
three aspects: internal consistency, temporal stability, and parallel
forms. Internal consistency is measured by correlation between elements
contained in the test. Temporal stability is not very applicable to the
case of color vision because majority of color vision deficiencies are
congenital and do not vary with time. Reliability of parallel forms, on
the other hand, is critical in this study. Reliability of several
parallel forms was calculated in this study: among two test forms (test
A and B) generated by two sets of random seeds for each subjects, among
two types of computer monitors (LCD and CRT), and among test and retest
performances. Specificity and sensitivity of the ATCOV test was
established by correlating the subject's performance scores with the
Dvorine test scores.

**Project Efforts**

***Data collection***

Subjects were recruited by advertising in local newspapers (such as
Norman transcript, Oklahoman Daily, and the Oklahoman), sending multiple
mass emails to all OU students (reaching about 21122 students),
providing course credit to students in certain classes, and posting
advertisement on various OU campus locations. The qualification of the
subject is male between the ages of 19 and 56.

The data were collected in two phases. The analysis results from the
phase one were considered by the researchers, and modifications were
made to the ATCOV. The modified tests were used during the phase two of
the project. The data collection efforts are summarized in Table 1, and
described as follows.

Table 1. Description of Data Collection Efforts

  ---------------- ------ ----- ------- -------------------------------------
                   CVD    CN    Total   Description

  Phase one-1^st^  69     20    89      Take test A long version, then test B
  stage                                 long version

  Phase one-2^nd^  13     6     19      Take test A long version, then test A
  stage                                 short version

  Phase two- 1^st^ 63     101   164     Take modified test A short version,
  stage                                 then test B short version

  Phase two- 2^nd^ 10     50    60      Take modified test A short version,
  stage                                 then test A short version again
  ---------------- ------ ----- ------- -------------------------------------

*Phase One*

In phase one, the long version of the ATCOV tests were used. The purpose
of phase one is to initially validate the ATCOV test by studying its
validity and reliability, and derive any needed modifications to the
test. The relationship between the test performance and the subject's
cognitive ability were also investigated. These two should be unrelated
to each other because ATCOV is not intended to test subject's ability to
perform complex tasks. In phase one, the ATCOV test usability were also
studied through questionnaires filled out by the subjects after taking
the tests. Two parallel test forms, test A and test B, were generated
using different computer random seeds which presented color patterns at
different locations on screen. The following data were collected during
each test session: the subjects' Dvorine test score, vision acuity, D-15
test score (tested twice), performance measures of the four tests in
both test A and B forms, subjective ratings of the test experience with
questionnaires, and subject's cognitive ability.

There were two stages of data collection in the phase one test. In the
first stage, subjects took the long version of test A and test B (either
on CRT or LCD screen). The long version test presents the ten colors
twice, where the first half tests ten colors, then the second half tests
all ten colors again. Such test design allows calculation of split-half
reliability. In the second stage, subjects took the long version of test
A, then the short version of test A, all on LCD monitors. The short test
version tests the ten colors for just once. The second stage allows
calculation of test stability by correlating the performance of the long
version with that of the short version.

The experimental procedure for the phase one test is described as
follows:

1\) Debriefing of experiment, fill out informed consent form and
demographic questionnaire;

2\) Administer DVorine test;

3\) Administer D-15 test twice;

4\) Test Tutorial and practice;

5\) Run long version of tests A (test1- test 4);

6\) Short break;

7\) In 1^st^ stage: Run long version of test B (test1-test 4);

In 2^nd^ stage: Run short version of test A (test 1-test 4)

8\) Fill out the usability questionnaire for test 1-4;

9\) Take the cognitive ability test;

10\) Fill out payment form.

The total length for the phase one test is around 3.5 hours for CVD
subject and 2 hours for CN subjects. Data from 82 CVD and 26 CN subjects
were collected, of which data from 63 CVD and 19 CN subjects were
collected in the first stage, and data from 12 CVD and 7 CN subjects
were collected in the second stage. Due to some errors in the data
collection process (e.g. not following experimental procedure precisely,
subjects not paying attention during test, or subject missing one test
trial etc.), data of some subjects were not usable or have some part
missing in the final calculation of results. Data from 76 color vision
deficient and 25 CN subjects were of the standard format as designed by
the experimenter,

*Phase Two*

During the phase two of the test, the modified version of ATCOV were
used. This version is also a short version which tests the ten colors
only once. This version was developed based on the data analysis from
the phase one test. Some modification was made to the colors used in the
test. Data were collected from 73 CVD and 151 CN subjects during phase
two test.

There were also two stages of data collection in the phase two test. In
the first stage, subjects took the parallel form test.  In the parallel
form test, subjects performed test A first, followed by test B. All
tests were performed on the LCD screen. 63 CVD and 101 CN subjects took
the first stage test. Data from this stage allows calculation of the
reliability of parallel forms for the modified test version. In second
stage, subjects took the test-retest form of the test. Subjects
performed test A first, then repeated test A again. All tests were also
performed on the LCD screen. 10 CVD and 50 CN subjects took the second
stage tests. Data from the second stage allows calculation of the
test-retest reliability.

The experimental procedure for phase two test is described as follows:

1\) Debriefing of experiment, fill out informed consent form and
demographic questionnaire;

2\) Administer DVorine test;

3\) Test Tutorial and practice;

4\) Run short version of tests A (test1- test 4);

5\) Short break (if needed);

6\) In 1^st^ stage, run short version of test B (test1-test 4);

In 2^nd^ stage, run short version of test A (test 1-test 4);

7\) Fill out payment form.

The total experimental time was around 45 minutes for the CN subjects,
and one hour and fifteen minutes for the CVD subjects.

***Data Analysis***

Data analysis programs were developed for all tests in MATLAB. The
performance measures of all four tests were based on the correctness of
the responses, not on the response time. Test1, Test2, and Test4 require
subjects to select multiple targets among many distracters. Hit rate and
false alarm rate were calculated for Test 1, 2, and 4. Hit rate was
calculated as the number of correctly clicked targets divided by the
total number of targets, and the false alarm rate was calculated as the
number of false alarms (wrong clicks) divided by the total number of
clicks. The overall performance measure for test 1, 2, and 4 was
calculated as hit rate minus false alarm rate. For test 3, the percent
of correctness rate was calculated as the number of correctly detected
target divided by the total number of test trials.

Based on the calculated overall performance of the four tests on CN and
CVD subjects, measures including specificity, sensitivity, and
reliability were derived. In the phase one experiment, we also
investigated issues including the relationship between the test
performance and subject's cognitive ability and the usability of the
test. To derive specificity and sensitivity, the cut-off score for the
tests needs to be determined. In order to pass the ATCOV test, a subject
needs to pass all three tests: test 1, test 2, and test 3 (performance
exceeding the cut-off scores). If a subject fails any of these three
tests, he can perform again with the parallel form of the test. Passing
the parallel form ATCOV test can still allow the subject to be
considered as color normal. Several cut-off scores were tried to obtain
the most desirable specificity values. The cut-off score should produce
specificity value of greater than 0.95. It needs to be refined based on
data from at least 100 CN subjects. After several rounds of trials, the
cut-off scores for test 1, test 2, and test 3 were set to 0.90, 0.75,
and 0.90 respectively.

The PI worked closely with the grant monitor to interpret the results
from the test, and made necessary changes to the ATCOV test when the
data indicated such need. For example, test 1 in the phase two was
modified to eliminate the magenta color because we noticed that subjects
were not familiar with the color name. We also noticed some sub-optimal
performance of the CN subjects. To fully motivate the subjects and
ensure active involvement of the subject in the test, we modified the
tutorial and instructions of the test. As a result, the performance of
the CN subjects in the phase two of the test was quite stable.

In addition to the general purpose computer programs for calculating
test results, several special purpose programs for data analysis were
also developed, including those to obtain split-half results of the
tests, those to calculate performance when ignoring certain colors in
tests (e.g. magenta color in test 1, dark green color in test 2), those
to obtain the confusion matrix for all colors, and those to obtain the
hit and false alarm rate for individual colors etc. Detailed results for
all data were sent to the grant monitor in Excel format. Throughout the
duration of the project, Powerpoint slides that summarized the results
were created periodically to report the project progress.

**Results**

***Reliability***

As shown in Table 2, three forms of reliability were calculated for the
test: split-half reliability between first and second half within the
test (obtained from phase one test data), reliability between parallel
forms of test A and B (obtained from stage one of phase two test data),
and test-retest reliability (obtained from stage two of phase two test
data). Most correlation coefficients were strong and significant. The
relatively low coefficient for parallel forms of test 4 was found for
the CN subjects, which might be due to the ceiling effect of test
performance.

Table 2. Reliability of the Four Tests

+----------------------------+-------------+------+------+------+------+
| Reliability                | Subject     | Test | Test | Test | Test |
|                            |             | 1    | 2    | 3    | 4    |
+----------------------------+-------------+------+------+------+------+
| Repeat within test         | CVD (n=82)  | 0.85 | 0.71 | 0.83 | 0.94 |
|                            |             |      |      |      |      |
| (based on phase 1-1^st^ &  |             |      |      |      |      |
| 2^nd^ stage data)          |             |      |      |      |      |
+----------------------------+-------------+------+------+------+------+
|                            | CN (n=26)   | 0.32 | 0.64 | 0.72 | 0.87 |
+----------------------------+-------------+------+------+------+------+
| Parallel form A and B      | CVD (n=63)  | 0.85 | 0.79 | 0.76 | 0.93 |
|                            |             |      |      |      |      |
| (based on phase 2-1^st^    |             |      |      |      |      |
| stage data)                |             |      |      |      |      |
+----------------------------+-------------+------+------+------+------+
|                            | CN (n=101)  | 0.50 | 0.36 | 0.61 | 0.04 |
+----------------------------+-------------+------+------+------+------+
| Test-retest                | CVD (n=10)  | 0.91 | 0.87 | 0.80 | 0.52 |
|                            |             |      |      |      |      |
| (based on phase 2-2^nd^    |             |      |      |      |      |
| stage data)                |             |      |      |      |      |
+----------------------------+-------------+------+------+------+------+
|                            | CN (n=50)   | 0.69 | 0.56 | 0.11 | 0.67 |
+----------------------------+-------------+------+------+------+------+

***Specificity***

Because the ATCOV test version used in the phase two test is the
modified version after considering results from phase one, and is
therefore closest to the final deployed test version, the results from
the phase two experiment are used for calculating the specificity and
sensitivity of the ATCOV test. Also, because the rule for determining
whether to pass a subject or not uses the parallel test forms, data from
the first stage of phase two with both test A and B were used to
establish the specificity and sensitivity.

Specificity of a test refers to the proportion of people without disease
who have a negative test result. In another word, it refers to the
proportion of CN subjects diagnosed with Dvorine test whose performance
measures were above the cut-off scores in the ATCOV test, and therefore
would be diagnosed as CN according to ATCOV. In stage one of the phase
two test, data were collected from a total of 101 CN subjects. When we
set the cut-off score to 0.90 for test 1, 0.75 for test 2, and 0.90 for
test 3, there were three CN subjects whose performance fell below such
cut-off score, and hence would be diagnosed as color deficient according
to the ATCOV test. Therefore the specificity of the ATCOV is
(101-3)/101=0.97.

***Sensitivity***

Sensitivity of a test refers to the proportion of people with disease
who have a positive test result. In another word, it refers to the
proportion of CVD subjects diagnosed with Dvorine test whose performance
measures were below the cut-off scores in the ATCOV test, and therefore
would be diagnosed as CVD according to ATCOV. In phase two, data were
collected from a total of 63 color deficient subjects. When we set the
cut-off score to 0.90 for test 1, 0.75 for test 2, and 0.90 for test 3,
there were one color deficient subjects who exceeded such cut-off score,
and hence would be diagnosed as CN according to the ATCOV test.
Therefore the specificity of the ATCOV test is (63-1)/63=0.98.

***Relationship between cognitive ability and test performance***

Based on data from phase one test, correlation between the cognitive
ability and test performances are -0.17, -0.08, -0.13, -0.28 for test 1,
2, 3, and 4 respectively. Such correlations are considered as weak.
Therefore, we know that ATCOV is not testing the subject's problem
solving skills on complex tasks.

**Conclusion**

Based on the data analysis results obtained through this project, the
ATCOV test is proven to be reliable with respect to internal
consistency, repeatability, consistency between parallel versions, and
testing monitors (LCD vs. CRT). The ATCOV tests also have high
sensitivity and specificity. With the current cut-off scores, ATCOV
produced a specificity of 0.97 and sensitivity of 0.98 based on the
results from 101 CNs and 63 CVDs.

The ATCOV test performance measures are not correlated with individual's
cognitive ability. It can be self-administrated or
instructor-administrated with minor trainings on instructors. The job
applicants with color deficiency can complete the test within 20 to 40
minutes. Such results support the deployment of ATCOV test in the job
selection of air traffic controllers.