during the sector validation conducted each year.
This assessment involves estimating the sector com-
plexity using a formula that weights various ATC
functions (FAA Order 7210.46). These are the func-
tions and their associated weights (in parentheses):
number of departures (5), number of arrivals (4),
number of radar vectored arrivals (2), number of en
route aircraft requiring control actions (4), number of
en route aircraft not requiring control actions (2),
number of emergencies (4), number of special flights
(3), and number of required coordinations (1). These
eight functions are evaluated, weighted, and totaled to
derive the sector complexity workload value. The
density rating is calculated by averaging traffic vol-
ume in each sector over a three day period, represent-
ing an Aclanta ARTCC 90" percentile traffic day.

4,2 Operational Error Data

Quality Assurance (QA) personnel at each facility
are responsible for gathering data and completing an
OE Report in accordance with FAA Order 7210.3
(Facility Operations and Administration). For the
purposes of this study, a number of fields from 103
OE reports from Atlanta ARTCC were coded and
entered into a data file. Those OEs where more than
one sector was involved (13), no final report was
available (4), or the error was attributed to an equip-
ment failure (1), were not included in this analysis.
This left a sample of 85 OEs, covering a three year
period from June 1992 to June 1995. For each error,
the following variables were available:

Report number.

Date.

Time.

Flight level.

How many controllers were charged.
Causal factors.

Type of sector (ultra high, high, or low).
Sector number.

Radar or non-radar controller charged.
10. Number of aircraft in the sector.

PEN AWAWNS

? Three measures of ATC complexity for Atlanta ARTCC sectors are

11. Estimated traffic complexity. ?

12. Vertical separation.

13. Horizontal separation.

14. Number of controllers working the sector.

15. Whether training was in progress.

16. Number of controllers working (including trainees).

17. Whether the sector was combined with another
sector.

18.

19.

20.

Whether any positions were combined.

Whether another facility was involved.

Whether the controller was aware that the error was
developing.

4.3, Combined Data Set

To examine sector differences that might have
contributed to OE incidence, the sector and OE data
were combined. The corresponding sector data were
attached to each line of OE data (given that each OE
occurred in a specific sector), A result of connecting
sector features with OEs was that sector data were
counted more than once in some analyses, given that
more than one error occurred in many sectors.

4.4 Sector Characteristics

This section reviews the sector data to provide basic
information about the characteristics of the Atlanta
ARTCC airspace. Selected descriptive information
on general characteristics of all sectors in the center is
included to provide a basis for the investigation of
sector factors that may be associated with OEs.

A histogram of total errors (1992 to 1995) for the
45 sectors in the facility is found in Figure 2. This
forms a positively skewed distribution with a mean of
1.9 errors, a standard deviation of 2.1 errors, and a
median of 2.0 errors. (Each bar in the histogram
represents the count of errors between the value of the
preceding category and the value of the labeled category.)

Figure 3 shows the distribution of sectors for size or
volume of airspace. The average volume was 30,517.1
cubic miles (cu mi) with a standard deviation of
51,242.6 cu mi and a median of 11,314.0 cu mi.
There was a wide range of sector sizes with the smallest

used in this report. These include the sum of the 16CF factors, a

complexity calculation from the 1995 Annual Review, and the estimated complexity of the traffic situation at the time of an OE.

12


Sector Volume (cu mi)

B

E

E

E

8

gE

1

Error Severity Goup

Hgh Bror

FIGURE 17. SECTOR VOLUME FOR OE ERROR GROUPS

33



g

Proportion of Sectors

3

low Heh
Sector Type

FIGURE 9. PROPORTION OF SECTORS AND OEs BY SECTOR TYPE

Number of OEs

FIGURE 10. DISTRIBUTION OF OEs BY TRAFFIC COUNT

29


TABLE 5. FACTOR ANALYSIS RESULTS FOR SECTOR CHARACTERISTICS

Sector Variable Factor 1/Factor 2|/Factor 3|Factor 4/Factor 5|Factor 6
[Variance Accounted For 24% 22% 10% 7% 7% 4%
Number of VORTACs -0.05 0.24 0.21 0.81 -0.27 0.01
Number of shelves 0.09 70.54 0.44 0.36 0.14 -0.20
[Average complexity 0.74 0.21 -0.12 0.09 0.27 0.27
[Average density 0.11 0.51 0.36 0.32 0.02 0.51
Traffic volume 0.63 0.50 -0.14 0.06 0.25 -0.16
Frequency congestion 0.62 0.48 -0.09 -0.20 0.12 0.11
Climbing/descending traffic| 0.84 -0.31 0.10 0.11 0.11 -0.13
Coordination 0.48 0.11 0.33 -0.19 -0.41 0.05
Multiple functions 0.83 0.13 0.30 -0.06 -0.15 -0.11
Hubbing 0.55 0.18 -0.38 0.12 0.51 -0.18
Number of intersections 0.34 0.45 0.37 0.45 -0.20 0.17
Intersecting flight paths 0.45 0.38 0.34 -0.01 ~0.22 -0.20
Complex routings 0.55 0.33 0.06 -0.05 -0.39 0.00
Miles of jetways -0.47 0.75 0.22 0.00 0.15 -0.13
Miles of victor routes -0.33 0.88 0.21 0.01 0.00 -0.09
Miles of other routes -0.26 0.86 0.25 0.05 -0.07 -0.14
Percent of cube 0.15 -0.16 0.33 0.25 0.57 0.05
Military traffic 0.13 0.21 0.68 -0.47 0.07 0.23
Restricted areas 0.10 -0.47 0.48 -0.23 0.31 0.21
Aircraft mix 0.18 0.73 0.07 0.44 0.04 -0.24
Required procedures 0.85 0.19 0.17 -0.08 -0.12 -0.21
Radar/radio coverage 0.12 0.36 -0.59 0.31 0.10 0.47
Cubic volume -0.51 0.61 0.25 0.19 0.03 -0.12
Sector size 0.29 0.61 0.24 0.08 0.47 -0.03
Sequencing and spacing 0.57 0.39 -0.32 0.06 -0.11 -0.09
Weather 0.66 0.29 0.11 -0.08 0.18 0.10

TABLE 6. PRIMARY CAUSAL FACTORS

Primary Causal Factor Number | Percent
Radar display: Inappropriate use of displayed data: Other 36 42
Communication error 15 18
Radar display: Misidentification: Climbed aircraft with similar call sign 14 16
Radar display: Misidentification: Failure to maintain lateral separation 4 5
Radar display: Inappropriate use of displayed data: Mode C 4 5
Coordination 4 5
Data posting: Computer entry 3 4
Data posting: Flight progress strip 2 2
Radar display: Misidentification: Overlapping data blocks 2 2
Unknown 1 1

38



including task design” (p. 44). Given these definitions,
the focus of this analysis is on contributing factors related
to airspace structure and traffic flow characteristics.

Redding (1992) reviewed FAA OE reports and
incident statements for 1989 and determined that a
failure to maintain situation awareness (SA) was the
likely cause of most errors. Errors typically occurred
in moderate complexity traffic conditions with eight
aircraft under control. Communication and coordi-
nation problems accounted for the greatest propor-
tion of errors. The misidentification or misuse of
radar data was the attributed cause of 37.6% of total
errors. The author recommended that specific in-
struction and practice in maintaining SA be added to
the current ATC training program.

Schroeder and Nye (1993) reviewed the FAA’s
Operational Error/Deviation (OED) database for the
years 1985 through 1988 to examine the relationships
between workload (traffic load and complexity) and
causal factors. As was found in other studies (Kinney,
et al., 1977; Stager and Hameluck, 1990; Stager, et
al., 1989), most OEs occurred under average or lower
traffic complexity conditions. Schroeder and Nye
(1993) found differences in ratings of complexity of
the traffic situation under which the OEs occurred
across ARTCCs but found it difficult to determine
whether these differences had any influence on OE
occurrence.

Some relationships were noted among error catego-
ries. When the radar display factor was involved in the
OE, statistically significant correlations were found
with communication and coordination factors, sug-
gesting that these three variables may be associated
with OE incidents. Overall, the radar display was
involved in 56.8% of OEs. Communication and
coordination were cited 29.7% and 29.6% of the
time, respectively, while data posting and relief brief-
ing were associated with 20.4% and 4.2% of errors,
respectively. The finding of coordination problems in
these data is similar to Schroedet’s (1982) earlier work
in this area.

Rodgers and Nye (1993) conducted a study to
relate the severity of OEs to air traffic controller
workload, as measured by the number of aircraft being
worked and the complexity level at the time of the
incident. A specific question was whether more severe

errors occurred during periods of high workload.
There was also interest in the reported causes associ-
ated with errors and whether aircraft flight profile and
altitude could be involved. Finally, investigations
were conducted to determine the underlying factors
leading to severe OEs.

The OED database for the years 1988 through part
of 1991 was used for this study. Average traffic load at
the time of an OE was 8.8 aircraft. It was found that
neither traffic load (workload) nor air traffic complex-
ity was related to the severity of OEs. The authors
noted that most OEs occurred when at least one
aircraft was in level flight and at least one was descend-
ing or ascending. However, most moderately severe
errors occurred when both of the concerned aircraft
were in level flight. This latter statement echoes
Siddiqee’s (1973) hypothesis that en route ATC er-
rors should result from a loss of horizontal separation
between aircraft at the same level.

Moderately severe errors were most likely to result
at flight levels less than or equal to 29,000 ft. Hori-
zontal, but not vertical separation, varied as a function
of error severity. Higher horizontal separation was
associated with errors where the controller had aware-
ness of the problem. There were no salient factors
found to explain the 15 severe OEs found in the data
analyzed.

Most of the databases containing reports on OEs
do not include an underlying analysis of the factors
causing the error. For example, although judgment
problems might be cited, it is not certain if this
resulted from sector design problems, a poor com-
puter-human interaction (CHI), poor controller train-
ing, or other issues. Fowler (1980) reviewed ATC
problems from a pilot’s point of view and echoed such
concerns. He noted that the National Transportation
Safety Board typically stopped short of fully examin-
ing the human factors problems associated with avia-
tion accidents. He suggested that some errors were
symptoms of an underlying system weakness. Some of
the human factors issues in the ATC system that relate
to the issue of ATC complexity included controller
failure to coordinate handoffs, noncompliance with
procedures, noncompliance with LOAs, use of inap-
propriate procedures, and failure of managers to ad-
vise controllers of procedure changes. Adverse weather


MIL

MIX

NOINTSX

OTHRTS

PROC

REST

RING

SHELF

SIZE

SQ_SP

AVG_COM_

P=

0172
45)
911

- 0332

45)
-828

0947
45)
+536

-0839

45)
-584

+5239

45)
-000

+2989

45)
-046

0476

45)
+756

.3271

45)
+028

21221

45)
+424

+0080

45)
+959

+4424

45)
+002

AVG_DEN_

(
P=

Correlation Coefficients

-0710
45)
+643

.2633
45)
-081

0768
45)
616

~4267
45)
-003

21773
45)
+244

+2839
45)
-059

0267
45)
- 862

-1986
45)
.191

0755
45)
+622

+3072
45)
+040

-1495
45)
2327

CLDC

-0260
45)
-865

~ 4821
45)
-001

+2984
45)
+046

+4620
45)
-001

+6650
45)
-000

+0556
45)
717

+2230
45)

= .141

+3279
45)
-028

+2558
45)

= .090

-3856
45)
-009

+3634
45)
.014

COOR

(
P=

(Coefficient / (Cases) / 2-tailed Significance)

2505
45)
-097

-0702
45)
647

+2602
45)
084

-1087
45)

= 477

+5227
45)
.000

.0797
45)

= .603

+0021
45)
+989

-3562
45)

= .016

+0442
45)
+773

+1413
45)

= .354

-1680
45)
270

" , "is printed if a coefficient cannot be computed

A2

CUVOL

(
P=

oA
W

mA
W

1182
45)
+439

+3913

45)
-008

+2963

45)
+048

-7272

45)
-000

+2464

45)

= .103
-0588

45)

= .701

+2451

45)
.105

-0216

45)

= .888

-1605

45)
292

-5719

45)
-000

-0320

45)
-835

FPAPCT

+1326
(45)
P= .385

+2453
( 45)
= .104

+2063
( 45)
P= .174

-.1144
( 45)
P= .454

-1099
( 45)
P= .472

~.1347
( 45)
P= .378

+2471
(45)
P= .102

-.2108
( 45)
P= .165

+2990
( 45)
P= .046

-1399
( 45)
P= .359

-.0826
( 45)
P= .590




- - Correlation Coefficients - -

FREQ FUNC HUB INTR JET MIL
TOTALOE +4943 +3280 -3453 +1375 -0374 0418
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .001 P= .028 P= .020 P= .368 P= .807 P= .785
TOTCOMX 6542 +8331 +6129 -5236 -.1994 +2320
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .000 P= .000 P= .000 P= .189 P= .125
VICTOR -1622 -.0895 -.0551 ~2845 .8879 -.0951
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .287 P= .559 P= .719 P= .058 P= .000 P= .534
VoL 6815 .5107 +6355 +3882 -1451 -.1795
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .000 P= .000 P= .008 P= .342 P= .238
VORTAC -.1057 -0149 -.0878 +2792 +2162 -.2835
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .490 P= .923 P= .566 P= .063 P= .154 P= .059

Wx 6244 ~4854 +3879 +3248 -.0322 +2020
45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .001 P= .008 P= .030 P= .834 P= .183

(Coefficient / (Cases) / 2-tailed Significance)

" |" is printed if a coefficient cannot be computed

AG


- - Correlation Coefficients - -

RTNG SHELF SIZE SQ_SP TOTALOE TOTCOMX
AVG_COM_ .3271 ~.1221 -0080 -4424 -4189 -6951
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .028 P= .424 P= .959 P= .002 P= .004 P= .000
AVG_DEN_ -1986 -.0755 +3072 .1495 +2108 +2521
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .191 P= .622 P= .040 P= .327 P= .165 P= .095
CLDC .3279 +2558 -.3856 -3634 .3259 7285
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .028 P= .090 P= .009 P= .014 P= .029 P= .000
COOR -3562 -.0442 -.1413 +1680 -1482 +4835
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .016 P= .773 P= .354 P= .270 P= .331 P= .001
CUVOL -.0216 -.1605 .5719 -.0320 -.1354 -.2623
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .888 P= .292 P= .000 P= .835 P= .375 P= .082
FPAPCT -.2108 +2990 -1399 ~.0826 .1073 +1533
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .165 P= .046 P= .359 P= .590 P= .483 P= .315
FREQ .3763 -.2077 .1444 +4325 -4943 +6542
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .011 P= .171 P= .344 P= .003 P= .001 P= .000
FUNC -5365 -0914 -.1416 +4313 +3280 - 8331
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .550 P= .353 P= .003 P= .028 P= .000
HUB +2158 -.0376 +1047 +4593 +3453 -6129
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .155 P= .806 P= .494 P= .002 P= .020 P= .000
INTR +3050 -.1514 -0096 -3193 .1375 -5236
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .042 P= .321 P= .950 P= .033 P= .368 P= .000
JET -.0112 -.3105 -6547 -.1510 -0374 -.1994
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .942 P= .038 P= .000 P= .322 P= .807 P= .189

(Coefficient / (Cases) / 2-tailed Significance)

". “is printed if a coefficient cannot be computed

Al0


REFERENCES

Arad, B.A. (1964a, May). The control load and sector
design. Journal of Air Traffic Control, pp. 13-31.

Arad, B.A. (1964b). Notes on the measurement of control
load and sector design in the Enroute environment
(Report No. AD-659035). Washington, DC: Fed-
eral Aviation Administration.

Bertelson, P., Boone, J.P., & Renkin, A. (1965). Vitesse
libre et vitesse imposee dans une tache simulant le
tri mechanique do la correspondence. Ergonomics,
8, 3-22.

Buckley, E.P., DeBaryshe, B.D., Hitchner, N., & Kohn,
P. (1983). Methods and measurements in real-time
air traffic control system simulation (Report No.
DOT/FAA/CT-83/26). Atlantic City, NJ: Fed-

eral Aviation Administration.

Couluris, G.J., & Schmidt, D.K. (1973). Air traffic
control jurisdictions of responsibility and airspace
structure. In Conference on Decision and Control,
4th Annual Symposium on Adaptive Processes (pp.
655-8). New York: Institute of Electrical and Elec-
tronics Engineers.

Danaher, J.W. (1980). Human error in ATC system
operations. Human Factors, 22, 535-45.

Davis, C.G., Danaher, J.W., & Fischl, M.A. (1963). The
influence of selected sector characteristics upon ARTCC
controller activities (Contract No. FAA/BRD-301).
Arlington, VA: The Matrix Corporation.

Empson, J. (1987). Error auditing in air traffic control.
Information systems: Failure analysis; Proceedings of
the NATO Advanced Workshop. (pp. 191-198).
New York: Springer Verlag.

Federal Aviation Administration. (1984). Establish-
ment and validation of en route sectors (DOT/
FAA/Order 7210.46). Washington, DC: Fed-

eral Aviation Administration.

Federal Aviation Administration. (1989). Air Traffic
Control. (DOT/FAA/Order 7210.65}). Washing-
ton, DC: Federal Aviation Administration.

Federal Aviation Administration. (1991). Facility Op-
erations and Administration (DOT/FAA/Order
7210.3k). Washington, DC: Federal Aviation

Administration.

45

Fowler, F.D. (1980). Air traffic control problems: A
pilot’s view. Human Factors, 22(9), 645-53.

Grossberg, M. (1989). Relation of Sector Complexity to
Operational Errors. In Quarterly report of the FAA
Office of Air Traffic Evaluations and Analysis. Wash-
ington, DC: Federal Aviation Administration.

Hurst, M.W., & Rose, R.M. (1978). Objective job
difficulty, behavioural response, and sector char-
acteristics in air route traffic control centres. Ergo-
nomics, 21(9), 697-708.

Kinney, G.C., Spahn, J., & Amato, R.A. (1977). The
human element in air traffic control: Observations
and analyses of the performance of controllers and
supervisors in providing ATC separation services.
(Report No. MTR-7655). McLean, VA: METREK
Division of the MITRE Corporation.

Langan-Fox, C.P., & Empson, A.C. (1985). “Actions
not as planned” in military air traffic control.
Ergonomics, 28(11), 1509-21.

McCoy, W.E., & Funk, KH. (1991). Taxonomy of
ATC operator errors based on a model of human
information processing. Proceedings of the 6th In-
ternational Symposium on Aviation Psychology (pp.
532-7). Columbus, OH: The Ohio State University.

McKinley, J.B., & Jago, R.J. (1985). Potential control-
ler error modes in automated en route ATC
(AERA). Proceedings of the 3rd International Sym-
posium on Aviation Psychology (pp. 189-97). Co-
lumbus, OH: The Ohio State University.

Mogford, R.H., Guttman, J.A., Morrow, S.L., &
Kopardekar, P. (1995). The complexity construct in
air traffic control: A review and synthesis of the
Literature. (Report No. DOT/FAA/CT-TN95/22).
Adantic City, NJ: Federal Aviation Administration.

Mogford, R.H., Murphy, E.D., Yastrop, G., Guttman,
J.A., & Roske-Hofstrand, R.J (1993). The applica-
tion of research techniques for documenting cognitive
processes in air traffic control. (Report No. DOT/
FAA/CT-TN93/39). Atlantic City, NJ: Federal

Aviation Administration.


significant ANOVAs (with p <.05). Ineach instance, the
only significant contrasts were between the zero and the
high-error sectors.

Number of major airports, although correlated
with OE group, was not include in the MANOVA
because the variable contained too few categories. It
was not possible to conduct a chi square test against
OE category because of zero frequency counts in some
cells of the contingency table.

Although sector size did not emerge as a discrimi-
nator for OE frequency groups in this analysis, a trend
noted in the data is worth reporting. Figure 17 shows
a graph of sector size as a function of OE group. The
variance evident within the no and low-error groups
(as represented by the standard deviation error bars)
was probably the reason that significant differences
were difficult to detect using ANOVA. However, it is
clear that the high-error sectors were distinct in hav-
ing aconsistently smaller volume than the zero or low-
error sectors, This is supported by a statistically
significant ¢ test (in spite of the high variability)
between the combined no and low-error groups versus
the high-error group, #(37) = 2.57, p = 014.

Another analysis of differences between sectors
with low and high OE frequencies was conducted to
consider relevant OE variables. The difference be-
tween this and the preceding analysis is that OE
characteristics recorded at the time of the error, as
opposed to general sector features, were being consid-
ered. The goal was to determine if OEs that occurred
in sectors with many errors were different from those
taking place in low OE sectors.

Table 11 shows the bivariate Pearson and Spearman
correlations (for only those variables where p < .1)
between sector OE frequency measures (number of
OEs and OE frequency groups) and relevant OE
variables from Section 4.2.

A MANOVA was not conducted, in that only one
variable had a significant correlation of p < .05 with
OE group. However, the univariate ANOVA for
number of aircraft in the sector was significant at p <
-1, with means larger for the high-error sector group
(M = 7.49 versus M = 8.55). The ANOVA conducted
for complexity was significant at p < .05. In the low-
error group, the mean for complexity was M = 3.05
and in the high-error group it was M = 3.64. Also, the

test involving workload index was significant at p <
-05, with means larger for the high-error sector group
(M = -.44 versus M = .41).

A chi square test for position combined revealed a
significant interaction between lowand high-error group
and whether the sector was combined with another
sector at the time of the error, x7(1) = 7.13, p = .008.
Positions were combined less frequently in sectors with
a high frequency of errors.

Two other techniques, multiple regression and
discriminant analysis, were employed to predict OEs
using the variables from Section 4.1. The relevant and
permissible sector characteristic variables identified
in Section 4.1 that had significant correlations with
number of errors were submitted, using a stepwise
procedure, into a multiple regression analysis with
total OEs as the dependent measure. (In this case,
Table 9 was not used for screening given that the
stepwise procedures screens for the contribution of
variables to the analysis.) The results shown in Table
12 suggest that it is possible to explain OE incidence
in the Atlanta ARTCC sectors by evaluating fre-
quency congestion and the influence of restricted
airspace. The amount of variance in OE incidence
accounted for by these two factors was 31%. (With three
outliers removed, this increased to 45%, p = .000.)

Discriminant analysis was used as an alternative
approach for predicting which sectors would have
errors. Similar to the MANOVA discussed earlier, it
determined how well the relevant and permissible
variables in Section 4.1 distinguished between no-
error, low-error, and high-error sectors. Discriminant
analysis has the advantage of providing success rates
for predicting group membership.

All of the applicable and permissible ATC com-
plexity variables were made available for the proce-
dure. Applying a stepwise approach yielded two
discriminant functions, with a combined 42(4) =
18.33, p = .001. The first function provided most of
the discriminating power; the second was not signifi-
cant when the first was removed. The first function
accounted for 33% of the variability in the grouping
data. Frequency congestion and the effect of restricted
airspace were the only two statistically significant
factors entered into the equation (p < .01).


mw Aware
Not Aware

S30 Jo sequNN

Sector Name and Total OEs

FIGURE 18. OE FREQUENCY FOR SA AND NO-SA ERRORS AS A FUNCTION OF TOTAL OEs

34


3.1
3.2
3.3
3.4

4.1
4.2
43
44
45
4.6

yee NN YP SP dD

Ye br Ss

TABLE OF CONTENTS

Page
PURPOSE

INTRODUCTION

SCOPC seresssssessssrecccsssessssnsesssssesssnsesessnasesssnsceasanecssssuscesssvscessseeesssssesssuseessssecsssusecsssusessnuteccsnsressansesssanessssnees 2
Summary of the Literature ......ccscsssscssssecsssseccssssecssneccssssecesnsesssusessesneesssnscesssnscessnsesssssecsssusessssecsesseses 3
Discussion ....essesssssssssessssssessesssessessessesssesssesessesesessvcsnessvesnessesssssesssssuesssavecsesarenseauessuesnecnessessncaserseesseess 9
Conclusions from the Literature Review ..

ANALYSIS OF OE DATA

Sector Dat ...scesssesseerseees
Operational Error Data ..s..sssssssesssesssseesssessseesstecsneesanenseeennvens

Combined Data Set..sssssssscsssssssscsssssssssessscssssusesssessssssvensssssssssessesesssnnueseeceessnneseseeesnnaneessessssneescesassssee 12
Sector Characteristics .....cs.ccssssccsssescsssessssssessssssesssssseesssuseesssuscessuseessnseesssueessnneesssnsessssnsecssssstsssseesssseee LZ
Operational Error Analysis .....ssssscssssssssssssssssssesscsssssesesssccssssveseessssssnnssssssssusessseneasnnetecsesennnesesesesenee LB
Summary and Discussion......sccsssssssssseesssssceesseesssnseesssssessssssesssssecesssusecssaveccesseccnaeessaneesssnsessssneceessee LZ

LIST OF FIGURES

Factors Affecting Controller Workload

Distribution of Sectors by Number of OEs...............

Distribution of Sectors by Cubic Volume of Airspace .

Distribution of Sectors by Type ....sssssssssseseesseesesnee
Distribution of Sectors by Average Traffic Demsity.....cessssssssssesesescsssseesssessnneeesseesens

Histogram of OE Count by Time of Day.n.sssssssssccssssssecsssssssssnsesssssssuerecctscsnesteceesssneseseessssneeseesessss 27
Distribution of OEs by Flight Level ....ssssssssssessecsecssessssssssssssssssssesvesssssseseseeeeeeeessenassnnsnsnsnsnssseses 28
Distribution of OEs by Sector Type ..sssesscssssssssssssscsssssessesessssssuesesessssnesesensuneesceessnnesssssssneseeesesnss 28
Proportion of Sectors and OEs by Sector Type ..ssssssssssssssssessssssssssssnssnsnesssissseseeeeceneseesssssssnssssansesse 29)
Distribution of OEs by Traffic Count.....
Distribution of OEs by Complexity Rating
Distribution of OEs by Workload Index...
Distribution of OEs by Vertical Separation......sscssccssssssessscssssssnerecsessssneeeeees

Distribution of OEs by Horizontal Separation .........csssssessssssssssessssssssssssesecesesnesesesecsnsneeesessssneeseeee 31

iii


Number of Sectors

4 5 6 1 8 9 10 ul
Average Traffic Density

FIGURE 5. DISTRIBUTION OF SECTORS BY AVERAGE TRAFFIC DENSITY

Number of OEs

Time of Day (hrs)

FIGURE 6. HISTOGRAM OF OE COUNT BY TIME OF DAY

27


being 4,695 cu mi and the largest at 212,370 cu mi. A
large proportion of sectors were between 5,000 and
15,000 cu mi.

As shown in Figure 4, most sectors were low (0 ft to
flight level [FL]230) or high (FL240 to FL340). Nine
sectors were ultra high, or FL350 and above.

Average traffic density (as measured during the
1995 facility survey) was 6.1 aircraft, with a standard
deviation of 1.7 aircraft and a median of 6.1 aircraft.
The distribution of sectors for average traffic density
is shown in Figure 5.

A factor analysis using sector variables was completed
to explore the underlying structure of sector characteris-
tics. The principal components extraction produced six
factors that accounted for 76% of the variance in the
measures. Table 5 shows the unrotated factor matrix
which was more easily interpreted than the varimax
rotation. (Variance accounted for is in the first row of
data. Highest variable loadings are in bold type.)

Fifty-six percent of the variance was explained by
the first three factors. The first factor appeared to be
related to traffic volume and activities associated with
managing aircraft. The annual facility review mea-
sure, average complexity, was loaded on this factor.
Average complexity is based on number of departures,
arrivals, en route aircraft, emergencies, special flights
and required coordination. Factor 1 also had loadings
for climbing/descending traffic, frequency conges-
tion, traffic volume, and other traffic management-
related factors. Accordingly, Factor 1 was called
“Traffic Activity.”

Factor 2 was named “Size” in that it had loadings
for cubic volume of airspace, miles of airways, and
sector size. Both the objective and subjective measures
of sector size were related to this factor. The presence
of a negative weighting of aircraft mix implied that a
low degree of mix (indicating a predominance of
larger, jet aircraft) was associated with larger (and
higher) sectors.‘ Miles of airways and sector shelving
were also part of this factor.

Factor 3 was concerned with military traffic activ-
ity and airspace. It also had a loading for adequacy of
radio and radar coverage. Although this third variable
did not seem to be directly related to military func-
tions, it may have been conceptually associated with
military airspace. Areas that have poor radio and radar
coverage, or are controlled by the military, are rela-
tively inaccessible to FAA ATC. Factor 3 was called
“Military.”

Factor 4 was only concerned with the number of
VORTACS and intersections. Factor 5 was not well-
defined, in that it only had a loading for percentage of
a regular shape (such as a cube) that the sector filled.
Factor 6 had the distinction of being associated with
average density, the second annual review measure.
The fact that this variable was not associated with
Factor 1 suggests that it may not actually be measur-
ing traffic activity but some other sector characteris-
tic. There were two variables, average density and
number of intersections, that were identified with
more than one factor.

4.5 Operational Error Analysis

General OE causality information was extracted
from the OE Reports. Then the sector and OE data
sets were analyzed to search for any patterns that
might help explain OE occurrence.

4.5.1 Operational Error Causal Factors

Each of the 85 OEs was categorized by primary and
secondary causal factors at the time of the error.
Tables 6 and 7 show the breakdown of primary and
secondary causes by number and by sector. (Only 22
errors were assigned secondary causal factors.)

Many of the errors (a total of 70%) were primarily
attributed to problems interpreting the radar display.
Eighteen percent of the errors were due to communica-
tion problems (including transposition, misunderstand-
ing, readback, and acknowledgment). Coordination and
data posting accounted for 5% and 6% of the OEs.

The multivariate procedures used in this study were often based on a relatively low number of cases. It is recognized that a larger data set
will be necessary before drawing fitm conclusions from such analyses. A correlation table for these variables is in Appendix A.

“Aone-way analysis of variance determined that there was a significant difference in sector size as a function of altitude level, F(2, 44) = 34.66,
p=.000). Tukey Honestly Significant Difference post hoc tests showed a significant difference between low ot high sectors and ultra high

sectors, with higher sectors being larger.


or high) and number of aircraft in the sector (at the
time of the error), complexity (as rated at the time of
the error), workload index, and whether the position
was combined at the time of the error. It was found
that the high-error sectors had more aircraft, were
more complex, and had higher workload. High-error
sectors were also less likely to have positions com-
bined than the low sectors.

Several approaches were used to attempt to predict
OE occurrence from sector and traffic variables. Us-
ing multiple regression, it was possible to account for
31% of the variance in the total sector errors using the
sector characteristics of radio frequency congestion
and effects of restricted areas. A discriminant analysis
using frequency congestion and the effect of restricted
areas resulted in a formula that was able to classify sectors
into error frequency groups with 58% average accuracy.

OEs were separated into those where the primary
controller had awareness of the developing problem,
as opposed to those where no awareness was present.
The only OE characteristic that was significantly
different between SA error groups was horizontal
separation which was greater when SA was present.

Using sector variables, significant differences were
found between SA and no-SA OEs for amount of
military traffic and the effect of military operating
areas, with no-SA OEs tending to occur with more of
each. As the frequency of errors within individual
sectors increased, there was evidence that SA dimin-
ished. In high-error sectors, there was a general reduc-
tion in awareness of errors. Thus, those sector
characteristics listed previously that discriminated
between low- and high-error sectors also probably
have a relationship to SA.

1.6.2 Relationships to Previous Research

The literature review identified many factors pos-
sibly related to OE or conflict occurrence. Table 4 has
been reproduced in Table 13 to list the findings from
this study that address the issues raised in the litera-
ture review. (NS indicates no statistically significant
results.)

Although it was possible to summarize the primary
and secondary causes attributed to operational errors
in the 1992 to 1995 data, the rating system had
changed so that direct comparisons to Kinney, et al.

(1977), Stager and Hameluck (1990), and Stager, et
al. (1989) were not feasible. However, Schroeder
(1982) noted that coordination was cited as a contrib-
uting factor in 27 to 54% of the OE reports between
1969 and 1980. Redding (1992) apparently used the
more recent classification system and found that
misidentification or misuse of radar data was cited in
38% of the 1989 error reports. Schroeder and Nye
(1993) employed the same categories for 1985 to
1988 data. Table 14 shows a comparison of the
Schroeder and Nye (1992) results with the 1992 to
1995 data.

A larger proportion of errors in the current study
were attributed to problems with the radar display
than in previous reviews. Communication problems
remained about the same as in the Schroeder and Nye
(1993) report, but fewer errors were assigned to prob-
lems with coordination or data posting.

Many factors were suggested in the literature re-
view as being possibly related to OEs (Arad, 1964;
Schmidt, 1976; Couluris and Schmidt, 1973; Stein,
1985; Buckley, et al. 1983; Fowler, 1980; Grossberg,
1989; Mogford, et al., 1993; Empson, 1987; Langan-
Fox and Empson, 1985). These are listed in Table 13.
The variables in the current study that were found to
be correlated with OE rate or that distinguished
between high and low OE sectors are linked, where
possible, to the specific issues raised in the literature
review. In many cases, there is some correspondence,
indicating that some of the findings from the current
research directly support (or in some cases fail to
corroborate) previous work in this area.

Several studies noted that most OEs occur during
times of moderate workload or traffic volume (Kinney,
et al., 1977; Schroeder, 1982; Stager and Hameluck
(1989, 1990); Redding (1992); Schroeder and Nye,
1993.) While no direct ratings of workload were
available for the current set of errors, there were
indications that errors occurred under higher than
normal traffic densities. The average traffic load at the
time of an OE was eight aircraft, the same number
found by Redding (1992). Rodgers and Nye (1993)
found that the average traffic load for 1988 to 1991
OEs was 8.8 aircraft. However, indirect corrobora-
tion of the findings regarding moderate workload
were found in the complexity ratings given each OE.


The horizontal and vertical separation data were
combined into one separation measure by calculated
minimum root mean square (RMS) distance for each
OE. Figure 15 shows the RMS distance for OEs that
occurred below and above FL290. The mean separa-
tion was 21782.8 ft, with a standard deviation of
5641.8 ft and a median of 22841.1 ft.

Figure 16 shows the number of controllers working
at the time of the error. One controller, in the radar
position, always works the sector. As traffic increases,
a second controller in the data position is added.
During extremely busy periods, an assistant controller
is assigned. As can be seen in the graph, most errors
occurred with one or two controllers working. How-
ever, this may merely reflect normal staffing patterns.

One of the items contained on the final report
prepared after the occurrence of an OE requires an
assessment of the involved employee’s awareness of
the developing error. This item has been on the OE
final reporting form for the past 14 years. After listen-
ing to the associated voice tape, interviewing the
involved controller, and reviewing the error with
SATORI, quality assurance (QA) specialists make a
determination as to the controller's awareness. Al-
though SATORI simplifies the formulation of this
judgment, most QA specialists find the answer rela-
tively easy to ascertain.

Typically, if either the control action to provide
separation was not issued in a timely manner, or no
control action was initiated, the controller is judged
to be unaware of the developing error. However, if the
controller actively attempted to provide separation to
the involved aircraft, although the control action was
either inappropriate or inadequate, the controller is
judged to be aware of the developing error. In 73% of
the cases in the OE data set, the controller was found
to be unaware. Further analysis of OEs with regard to
controller SA is found in Section 4.5.4.

4.5.3 Accounting for Error Frequency

One of the primary goals of this project was to
determine if sector or traffic characteristics could
account for OE incidence. Three techniques based on
the general linear model were used to explore these
relationships. First, MANOVA was applied to deter-
mine if differences existed between sectors with no
errors, few errors, or many errors.

In Figure 2, the distribution of errors in the 45
sectors in the Atlanta ARTCC was examined. With a
mean of 1.9 errors and a standard deviation of 2.1
errors, it was decided to separate the sectors into OE
frequency groups with 0 errors, less than 4 errors (low-
error sectors), and 4 or more errors (high-error sec-
tors).’” There were 15, 22, and 8 sectors in each group,
respectively,

Bivariate Pearson and Spearman correlations were
calculated between 29 variables from Section 4.1 and
the sector OE frequency measures (number of OEs
and OE frequency group). All correlations significant
at the p < .1 are listed in Table 9.*

The correlation results were used to screen vari-
ables for inclusion in the MANOVA, with OE fre-
quency group as the independent variable. Eight
variables achieved statistical significance of p = .05 or
less. These were included in the MANOVA which was
significant with Hotellings F(16, 68) = 1.60, p =
-094°, Four dependent variables emerged with statis-
tically significant contributions (p < .05) to this result.
Four other variables were significant at the p < .1 level.

Table 10 shows one-way analysis of variance (ANOVA)
tests for the eight OpenCreate, 16CF, and facility review
data variables that were included in the MANOVA.
Results where p < .05 indicate statistically significant
differences in sector characteristics between error groups
are shown in bold print. (There were also several tests
significant at the p < .1 level.) Tukey Honestly Signifi-
cant Difference post hoc tests were performed on the

The division between low and high error sectors was set at one standard deviation from the mean, or at 4.0 errors.

"Results at or near the p = .1 level were reported in these analyses, given that the emphasis was on data exploration. In this case, a Type I error
(incorrect detection of a difference) would only lead to further investigation with a larger data set. A complete Pearson correlation table for
the sector variables in Section 4.5 is found in Appendix A. A few of the variables did not meet the assumptions of normality and

homoscedasticity, so the results should be interpreted with caution.

° Only those variables correlated with p< .05 were included in the MANOVAs. This was to control the number of variables in the analysis

and to improve the reliability of the outcome.


20
Q 15
3
ro
3
—
Z 10
5
o4
50 100 150 200 250 300 350
Flight Level (ft X 100)
FIGURE 7. DISTRIBUTION OF OEs BY FLIGHT LEVEL
45
40
35
30
3
2 25
5
=
Zz
15
10 8
=)
0

High Ultra High
Sector Type

FIGURE 8. DISTRIBUTION OF OEs BY SECTOR TYPE

28



Several authors noted the likelihood of OEs as a
function of phase of flight. Siddigee (1973) predicted
that most OEs would occur as a result of a loss of
horizontal separation between aircraft flying at the
same altitudes. In partial support of this claim, Kinney,
et al. (1977) found that most errors in SEIS data
happened in level flight and Rodgers and Nye (1993)
observed that moderate errors for 1988-90 OEs usu-
ally involved aircraft in level flight. However, Rodgers
and Nye (1993) discovered that most errors were
between an aircraft in level flight and one that was
climbing or descending.

All of the studies that focused on a review of error
database information found that OEs occurred under
moderate or low workload conditions. Rodgers and
Nye (1993) suggested that, theoretically, SA might
decrease under high workload conditions. However,
given the lower number of high workload OEs, it may
be the case that SA is enhanced as workload builds, but
decreases (perhaps due to fatigue) as workload subse-
quently diminishes (as also mentioned by Rodgers
and Nye, 1993). Low workload may not foster good
controller awareness due to the marginal cognitive
arousal level required.

Several of the studies reviewed were either theoreti-
cal in nature (Arad, 1964; Schmidt, 1976; Couluris
and Schmidt, 1973), were experiments designed to
investigate controller workload (Stein, 1985) or per-
formance (Buckley, et al. 1983), or were analytical
(Fowler, 1980) but have information applicable to
our topic, Arad (1964) developed mathematical mod-
els to predict conflicts and used rules of separation,
average traffic speed, number of aircraft under con-
trol, sector size, and flow organization as variables.
Schmidt (1976) took a similar approach and stated
that conflicts could be accounted for by flow rate,
separation standards, route geometry, aircraft speed,
aircraft flow rate, angle of airway intersection, and
number of flight levels. Couluris and Schmidt (1973)
suggested that controller actions (such as handoffs,
coordination, and pointouts) are affected by the exist-
ence and shape of sectors. Stein (1985) found that
controller workload was related to clustering of air-
craft in a small amount of airspace, number of hand-
offs outbound/inbound, and total number of flights
handled. Buckley, et al. (1983) discovered that traffic

10

density and sector geometry interacted to affect con-
troller performance. Fowler (1980) thought that ATC
complexity was affected by LOAs and weather.

These variables could be considered as ATC com-
plexity factors in the same vein as those suggested by
Grossberg (1989) and Mogford, et al. (1993). When
combined together, a substantial list of factors is
suggested that can be used in research on ATC com-
plexity, controller workload, and OE occurrence.
These variables were used as reference points, or
guides, while analyzing the characteristics of Atlanta
ARTCC sectors where OEs occurred.

Itis worth mentioning the work by Empson (1987)
and Langan-Fox and Empson (1985) as a different
approach to the previously discussed studies. Rather
than reviewing historical data, constructing math-
ematical models, speculating on causes, or conduct-
ing experiments, these authors collected real time data
by observing military controllers. They suggested
some of the now familiar factors that could affect
controller workload, such as airspace structure, proce-
dural demands, and traffic type. However, they dis-
cussed another interesting variable in noting that
errors might occur more frequently when controllers
cannot control their pace of work. It may be worth
gathering data on whether sector task load in OE
sectors is largely driven by external events or can be
modified or time-sequenced by the controller.

Finally, further research on OE occurrence should
attempt to explain the findings of Rodgers and Man-
ning (1995). It may be that ATC complexity factors
can help account for the observed increases in sector
transit time, handoffacceptance latency, vertical sepa-
ration, and aircraft density and the decrease in handoff
acceptance rate associated with operational errors.

3.4 Conclusions from the Literature Review
Although considerable work has been completed to
attempt to understand the causes of OEs in en route
airspace, much of this research has been limited to
using the available information contained in error
databases. Unfortunately, this approach restricts the
investigator to the data contained in the records and
precludes in-depth study of the perceptual, cognitive,
or environmental factors originally at play. The devel-
opment of SATORI has made it possible not only to


- - Correlation Coefficients

MIX NOINTSX OTHRTS PROC
MIL +0445 .1858 -.1221 .1406
( 45) ( 45) ( 45) ( 45)
P= .772 P= .222 P= .424 P= .357
MIX 1.0000 4975 -.6258 -0499
( 45) ( 45) ( 45) ( 45)
P=. P= .001 P= .000 P= .745
NOINTSX 4975 1.0000 -.3431 .1686
( 45) ( 45) ( 45) ( 45)
P= .001 P=. P= .021 P= .268
OTHRTS -.6258 -.3431 1.0000 0495
( 45) ( 45) ( 45) ( 45)
P= .000 P= .021 P=. = 747
PROC +0499 -1686 0495 1.0000
( 45) ( 45) ( 45) ( 45)
= 745 P= .268 P= .747 Ps.
RAD -.2040 -.1741 +1064 -0028
( 45) ( 45) ( 45) ( 45)
P= .179 P= .253 P= .487 P= .986
REST +2625 +2560 -.4039 -0180
( 45) ( 45) ( 45) ( 45)
P= .082 P= .090 P= .006 P= .907
RTNG -.1618 .1139 +2106 -5460
( 45) ( 45) ( 45) ( 45)
P= .288 P= .456 P= .165 P= .000
SHELF +5947 .5237 -.3032 +0297
( 45) ( 45) ( 45) ( 45)
P= .000 P= .000 P= .043 P= .846
SIZE - 3966 -.3113 -6077 -.1728
( 45) ( 45) ( 45) ( 45)

P= .007 P= .037 P= .000 P= .256
SQ_SP -.1417 -.0713 +0907 +5229
( 45) ( 45) ( 45) ( 45)
P= .353 P= .641 P= .554 P= .000

(Coefficient / (Cases) / 2-tailed Significance)

". "is printed if a coefficient cannot be computed

A8

wn

On
a

W

"

mwa

RAD

-.4327

45)
-003

-2040

45)
.179

217421

45)
+253

1064

45)
- 487

-0028

45)
-986

-0000

45)

4421

45)
+002

+2088

45)
-169

+4060

45)
006

0071

45)

= .963
-3489

45)
019

wn

yA

REST
+5929

45)

= .000
+2625

45)
-082

+2560

45)
+090

~4039

45)
006

-0180

45)
+907

4421

45)
-002

-0000

45)

.1153

45)
+451

+2873

45)
-056

-1132

45)
+459

+2600

45)
085


As shown in Figure 1, ATC complexity is the
underlying driver of controller workload. The proce-
dures required in the sector, flight plans, traffic load,
weather, and other variables form the basis for the
tasks the controller must complete. Although evi-
dence is weak, controller workload is probably associ-
ated with OE commission. As task information
processing requirements reach and exceed controller
sensory and cognitive capacities, aircraft may not
receive sufficient attention and control to maintain
required separation. Workload may be increased
through the presence and interaction of several com-
plexity factors that create competition for similar
cognitive resources. Alternately, isolated ATC com-
plexity factors may lead to unsafe conditions by plac-
ing focused demands on the controller. Such factors
may be transitory or sustained and may pose undue
strain on specific information processing channels or
capabilities (such as memory). For example, the manage-
ment of a sector may require the application of many
required procedures. Forgetting to apply these at the
correct time could lead to traffic problems, and errors.

The amount of workload experienced by the con-
troller is affected by the information processing strat-
egies adopted to accomplish required tasks. Such
techniques may have been learned in developmental
training or evolved on the job and may vary in effec-
tiveness. The influence of a complex ATC environ-
ment on workload can be ameliorated through the use
of strategies that maintain safety through, for ex-
ample, simpler or more precise actions.

Also relevantis the effect of equipment on workload.
The controller’s job will be made easier if a good user
interface is provided. This will ensure that adequate
and accurate information is presented to support
efficient task completion. Automation tools to sup-
port essential tasks should also be available. It is for
this reason that the FAA is developing decision sup-
port systems to facilitate more effective control ac-
tions.

Workload can also be influenced by personal vari-
ables, suchas age, susceptibility to anxiety, and amount
of experience. Variations in skill between controllers
can be quite pronounced. These factors can have a

strong effect on the workload experienced by a given
controller in response to a specific array of ATC
complexity factors.

The goal of this research was to isolate those ATC
complexity factors that create the conditions for OEs.
This is not to minimize the possible effects of other
CHI and operator-specific factors. Rather, it is to
begin to fill a research void on ATC errors concerning
the effects of the controller’s work domain: air traffic,
airspace, and their characteristics.

The paper is divided into two sections. The first
contains a brief literature review on OEs and their
causes. The second section discusses an analysis of
data available on OEs from the Atlanta ARTCC and
their relationship to sector and traffic characteristics.

The research was exploratory in nature and con-
sisted of a variety of analyses intended to map rela-
tionships in the data. Exploratory data analysis focuses
on generating theoretical models from data, rather
than testing a pre-existing model. By exploring the
data with an inquisitive mind, it is possible to discover
what is not necessarily expected. Visual representa-
tions (such as graphs) and the use of statistics that are
resistant to the effects of departures from assumptions
(such that variables are normally distributed) are
used. However, one outcome of using several statisti-
cal tests with the same data set is an increased chance
of finding statistically significant differences where
there are none. As a result, the statistical findings in
this report should be considered as tentative pending
a larger-scale study.

3. LITERATURE REVIEW

3.1 Scope

This review examines the research completed to
date on the causes of OEs. Literature sources included
the FAA Technical Center Library, FAA Headquar-
ters Library, FAA CAMI Library, PsychINFO (an on-
line database service of Compuserve), National
Technical Information Service database, and Embry-
Riddle Aeronautical University Library. Keyword
searches were conducted to identify publications con-
cerning ATC operational errors and sector design.


SOURCE FACTORS

ATCOOMPLEXITY:
AIRTRAFFIC PATTERN
AND SECTOR
CHARACTERISTICS

FIGURES

MEDIATING FACTORS

COGNITIVE
STRATEGIES

USER
INTERFACE

INDIVIDUAL
DIFFERENCES

RESULT

CONTROLLER
WORKLOAD

FIGURE 1. FACTORS AFFECTING CONTROLLER WORKLOAD

Number of Sectors

4

6
Number of OEs

FIGURE 2. DISTRIBUTION OF SECTORS BY NUMBER OF OEs

25



- - Correlation Coefficients

VICTOR VoL VORTAC Wx
TOTALOE -.0241 +3338 -.3050 -4493
( 45) ( 45) ( 45) ( 45)
P= .875 P= .025 P= .042 P= .002
TOTCOMX -.0382 +7066 -0157 - 6685
( 45) ( 45) ( 45) ( 45)
P= .803 P= .000 P= .918 P= .000
VICTOR 1.0000 +2232 +2596 -.0013
( 45) ( 45) ( 45) ( 45)
P=. P= .141 P= .085 P= .993
voL +2232 1.0000 - 0238 .5155
( 45) ( 45) ( 45) ( 45)
P= .141 P=. P= .876 P= .000
VORTAC +2596 +0238 1.0000 - .0232
( 45) ( 45) ( 45) ( 45)
P= .085 P= .876 P=. P= .880
Wx -.0013 -5155 -.0232 1.0000

( 45) ( 45) ( 45) ( 45)
P= .993 P= .000 P= .880 P=.

(Coefficient / (Cases) / 2-tailed Significance)

“, “is printed if a coefficient cannot be computed

AlS


It is also interesting to note that three of the
variables that were statistically significant, and all four
of the marginal variables were part of the Traffic Activity
factor. Thus, this factor alone may account for many of
the effects that contribute to OE occurrence.

One of the potential benefits of collecting data on
factors that might be related to OE occurrence is that
the ability to predict these events could improve.
This, in turn, would pave the way for the development
of practical tools for conflict management that might
be used in en route ATC operations. Several tech-
niques were applied to explore this possibility, all
using the same underlying statistical approach.

Multiple regression, which was used to build the
optimal mathematical combination of variables that
will predict the number of OEs in a given sector,
accounted for about 31 percent of the available infor-
mation. The size of this correlation is respectable on
theoretical grounds in that it demonstrates that there
is a firm relationship between sector characteristics
and OE rate. However, it is not of much practical
importance in that the equation, in its present form,
will not permit accurate conflict prediction.

The fact that only two variables (frequency conges-
tion and restricted areas) entered the regression analy-
sis suggests that there may be a significant amount of
redundant information in the data set. (This can been
seen in the correlations in Appendix A.) It is interest-
ing to note that one of these variables, frequency
congestion, also appeared in the multiple correlation
with overall ATC complexity derived by Mogford, et
al. (1993). The two variables were members of the
Traffic Activity and Military factors, respectively.

Although the regression analysis ultimately em-
ployed only two variables, this does not necessarily
imply that these are the only meaningful factors for
further study. The other sector characteristics also
have useful information, but their inter-correlations
suggest that there may be only a few underlying
themes, as demonstrated in the factor analysis. If
description of sectors is the goal, these variables should
be retained, for they provide a richness of detail. For
predictive purposes, however, it may be adequate to
employ a subset of the original measures.

22

Discriminant analysis was applied in an attempt to
use the available variables to predict whether a sector
belonged to the no, low, or high-error groups. Using
the original measures, frequency congestion and re-
stricted areas again emerged as the only two emergent
factors, and it was possible to achieve an average
classification accuracy rate of 58%. This resulted in a
success rate of 40%, 68%, and 63% for the no-, low-, and
high-error groups, respectively. The chance rates of
correctly identifying sectors from each group would
be 33%, 48%, and 18%, respectively, for the no-,
low-, and high-error groups. Thus, the discriminant
analysis adds predictive power, especially for identify-
ing potential high-error sectors.

Practically speaking, the approach reflected in cor-
relation or multiple regression might be the most
useful in ATC operations. The results of such analyses
would indicate that OEs would be more likely to
occur with the increase or decrease in certain dynamic
sector or traffic-related factors. Such information
might assist flow controllers and area supervisors in
taking steps to avoid problems before they develop.
Another application could be to assist with defining
such free flight concepts as dynamic density and
flexible resectorization.

There were also static measures, such as sector size,
that contributed toward high error frequency. How-
ever, from experience, facility personnel already are
familiar with which sectors are error-prone. Being
able to group sectors by error frequency, as predicted
by a set of sector and traffic measures, might be more
useful for airspace reconfiguration projects. Another
application could be to assist with dynamic
resectorization, a concept proposed for free flight.

It would be desirable to determine when high-risk
situations are developing and predict potentially prob-
lematic sectors. Using information on sector charac-
teristics, the methods used here show some promise,
but are not as yet sufficiently powerful. The 16CF
questionnaire needs further validation to ensure that
it is performing as intended. Some evidence of con-
current validity for some of the items was found in the
factor analysis. It would be interesting to create some
additional complexity questionnaire items for evaluation


EXECUTIVE SUMMARY

A study of air traffic control (ATC) complexity
issues associated with the causes of operational errors
(OEs) in the National Airspace System was con-
ducted. It consisted of a comprehensive survey of the
literature on OEs and a detailed analysis of OE data
from the years 1992 through 1995 from Atlanta’s Air
Route Traffic Control Center (ARTCC). Of specific
concern was the influence of airspace sectorization
and traffic characteristics on OE incidence. The Sys-
tematic Air Traffic Operations Research Initiative
(SATORI) system and other methods were used to
collect data for the analysis.

The literature review was conducted on the rela-
tionship of airspace and air traffic factors to OE
occurrence. Papers were analyzed for information that
might help identify sector characteristics or aircraft
flow patterns (also known as ATC complexity factors)
that could lead to a loss of separation between aircraft.
It was found that very little research has directly
addressed the issue of ATC complexity as a factor in
OE occurrence. A possible reason cited was that most
prior investigation has focused on the analysis of OE
database information that is lacking in specific details
on causality. Nevertheless, there was evidence that
sector and traffic characteristics could influence
workload and error incidence.

A theme in the literature emerged with regard to the
relationship of sector design and the amount of coor-
dination controllers must perform. Many studies cited
coordination problems as a factor in OEs. Additional
issues from the literature review were evaluated for
their role in creating the conditions for OEs. These
data came from both theoretical and empirical re-
search studies. The information extracted from the
literature review was used to guide the research at the
Atlanta ARTCC.

Data from SATORI, a complexity factor question-
naire, the 1995 facility review, and OE reports were
combined in a database. Descriptive information was
computed for error causality and sector characteris-

tics. A factor analysis of the sector variables yielded six
dimensions, the most important being the first three:
Traffic Activity, Size, and Military. OE conditions
were described as a function of time of day, flight
level, traffic density, complexity, vertical and hori-
zontal separation, severity, workload, number of con-
trollers working, and situation awareness (SA).

The 45 sectors in the Atlanta ARTCC were catego-
tized into No-, Low-, and High-Error groups. The
data collected on sector and traffic flow characteristics
were used to explore differences between groups that
might account for OE incidence. Fifteen variables
had statistically significant (or marginally significant)
correlations with error frequency. Further analyses
indicated that four variables (Weather, Radio Fre-
quency Congestion, Total Complexity, and Average
Complexity) were higher for the High-error group as
compared to the Low-error group. Sector size was
smaller for the High-error group, as compared to the
combined No- and Low-error categories. Four other
variables showed marginally significant differences
between groups.

Using several related statistical techniques, it was
possible to demonstrate that overall OE rate or sector
severity group could be predicted by a subset of the
sector characteristic and traffic flow variables. How-
ever, although there was statistical significance, accu-
racies were generally low, indicating insufficient power
for practical applications. Nevertheless, a firm theo-
retical relationship was demonstrated between sector
complexity and OE occurrence.

The data were also analyzed to define relationships
between sector and traffic flow characteristics and
controller situational awareness at the time of the
error. The only statistically significant variable was
horizontal separation; more horizontal separation was
found for errors where SA was present. Errors where
controllers reported that they were not aware of the
impending loss of separation occurred when there
were more military operational restrictions. Also, as


yee N AYR YD

ee ee
ae bE S

LIST OF FIGURES (continued)

Minimum RMS Separation for OEs
Number of Controllers Working at the Time of the Error ........ssssssssssssssssssssssesessnsesseesnsssneseseeessnee 32

Sector Volume for OE Error Groups

OE Frequency for SA and No-SA Errors as a Function of Total OES .....sssssesssssssssssessssssesssserecsssees 34

LIST OF TABLES

Number of Operational Errors as a Function of Traffic Volume and Workload Complexity
in 1974-76 SEIS Data.ssecsccssssssssssssssssssessescssssnseesscesssssessecessssuneceessssssuessesssnnsseeseensnnnaeesessaneeseeeeees 35

Measures Used in the Rodgers and Manning (1995) Study .....scssssssccsssssecssntecesnseescsnesssneesssneeeesaee 3D,
MANOVA Results «...scsscsssessssseescessesssssveecsssseecssssssssssscsssssecsssusecessusesssnsecessuneessnseesssuecessnsesessnecessseeeeass 35

Summary of Literature Review «0.0.0.0

Factor Analysis Results for Sector Characteristics ....sssssssssssssssesesssesssssseesssssssseesssessssneesesecenneeeseseeee 38

Primary Causal Factors ..

Secondary Causal Factors ..

Summary of all Causal Factors. ......
Correlations of Sector Variables with Number of Sector OEs and OE Group .....ssssssssssseeeeeees 39
ANOVA Tests for Differences Between Sector Characteristics .....s...scsssssssescsssseesssessssnsesesssssseeees 40

Correlations of OE Characteristics with Number of Sector OEs and OE Frequency Group ........ 40
Regression Analysis of OE Data ........ssssssssees +40
Relationship of Findings to Literature Review .. Al

Comparison of OE Causes

Appendix: Pearson Correlation Matrix of Sector Variables.........scsccsssssssessssseesssssecessnsessseessssssesssectsseeee AL


recreate OE incidents, but also to collect data on
controller activities, sector characteristics, and traffic
patterns during the time of the error. It is anticipated
that these tools will allow a deeper analysis of the
conditions surrounding OE occurrence.

The purpose of this literature review was to locate
research relevant to the relationship of sector charac-
teristics and traffic flow to OEs. It was found that
little work had been done to directly address this issue,
perhaps due to the limitations inherent in the avail-
able data. However, there were several studies that
made a case for the effect of ATC complexity factors
on controller workload or error frequency. In addi-
tion, it was possible to generate hypotheses about the
types of errors that might occur and factors that could
contribute to increases in workload, complexity, and
the probability of OEs. These findings were applied
while investigating the error data available from the
Atlanta ARTCC.

The confirmation that some of the variables iden-
tified in this literature search are related to OE occur-
rence is only the beginning of an effort to identify the
relevant human performance issues. Hopefully, fur-
ther research will permit the development of a human
error model that is founded on factual information
available from the system. This model should contain
hypotheses about how these variables are perceptually
and cognitively processed by the air traffic controller
to result in an error-prone situation.

4, ANALYSIS OF OE DATA

To more fully investigate the relationship of sector
characteristics and traffic flow to OEs, and to explore
some of the predictions and assumptions identified in
the literature review, sector and OE data from Atlanta
ARTCC were analyzed.

Information on sector characteristics was collected
using the SATORI OpenCreate application, a ques-
tionnaire based on the previous ATC complexity
work by Mogford, et al., (1993), and 1995 Atlanta
ARTCC facility review data. A set of OE data was
compiled from Atlanta ARTCC OE Reports. The
following sections specify the variables collected and

discuss the results of analyses of these data. The data
were divided in two sets, focusing separately on sector
and OE characteristics.

Given the exploratory nature of these analyses,
results were considered significant if p < .1. (Prob-
abilities that rounded to .1 were included.) Frequency
count variables were considered to be acceptable for
inclusion in parametric tests if they contained suffi-
cient categories and could be assumed to be normally
distributed in the population. Measures that had five
or less categories were not included and, if it was
indicated, were analyzed using non-parametric tests.

4.1 Sector Data

Data were collected on the characteristics of sectors
at the Atlanta ARTCC and were obtained from three
sources. First, OpenCreate was used to extract details
on the following variables for each sector:

Number of major airports.

Percentage of volume the sector occupies of a cube
or other regular geometric shape.'

. Number of shelves in the airspace.

Total cubic volume.

. Number of VORTACs.

. Number of obstructions.

. Number of intersections.

Nr

3
4.
5.
6.
7.
8. Miles of victor routes.

9. Miles of jet routes.

10. Miles of other routes.

During a visit to the Atlanta ARTCC, further data
were collected on the characteristics of sectors using a
questionnaire containing 16 ATC complexity factors
(16CF) adapted from Mogford, et al. (1993). One
Airspace and Procedures Specialist from each area in
the facility rated all of the sectors in his or her area on
all 16 variables (using 7-point scales). The total com-
plexity score was calculated by summing the 16 factor
ratings for a particular sector.

In addition, the results of the 1995 facility review,
consisting of average complexity and density ratings
for each sector, were included. The facility average
complexity estimate is calculated by facility personnel

Many sectors have a very irregular, three-dimensional shape. This measure helps estimate sector shape complexity by determining how close
the sector comes to the shape of a regular geometric object, such as a cube.

11


THE RELATIONSHIP OF SECTOR CHARACTERISTICS
TO OPERATIONAL ERRORS

1. PURPOSE

A research project was conducted on sector design
and operational errors (OEs) in en route air traffic
control (ATC) sponsored by the Federal Aviation
Administration’s (FAA’s) Civil Aeromedical Institute
(CAMI). The goal of the study was to employ the
Systematic Air Traffic Operations Research Initiative
(SATORI) system and other sources of data on en
route ATC sectors to investigate possible underlying
causes of OEs in the Atlanta Air Route Traffic Con-
trol Center (ARTCC).

2. INTRODUCTION

In the history of the FAA, no aircraft have collided
while under positive control in en route airspace.
However, aircraft have violated prescribed separation
minima and approached in close proximity. This
event can occur as a result of either a pilot deviation
or an OE. An OE takes place when an air traffic
controller allows less than applicable minimum sepa-
ration criteria between aircraft (or an aircraft and an
obstruction). The number of OEs is a primary index
of National Airspace System (NAS) safety.

Standards for separation minima are described in
the ATC Handbook (FAA Order 7110.65, and supple-
mental instructions). While there is considerable com-
plexity in those standards, at flight levels between
29,000 feet (ft) and 45,000 ft, Air Traffic Control
Specialists (ATCSs) at en route facilities are required
to maintain either 2,000 ft vertical separation or 5
miles (mi) horizontal separation between aircraft. At
flight levels below 29,000 ft with aircraft under in-
strument flight rules, ATCSs are required to maintain
either 1000 ft vertical separation or 5 mi horizontal
separation.

Immediately after the detection of an OE, a de-
tailed investigation is conducted in an attempt to fully
describe the events associated with the error’s occur-
rence. This includes removing the ATCS(s) from the

operating position and obtaining a statement from
each of the involved personnel, gathering the relevant
data (voice and computer tapes), and reviewing in
detail the events associated with the error’s occur-
rence. At the Atlanta ARTCC, the SATORI system is
used to re-create the error situation in a format much
like the one originally displayed to the ATCS (Rodgers
and Duke, 1993). SATORI allows for a more accurate
determination of the factors involved in the incident.
Once the OE has been thoroughly investigated, an
OE Final Reportis filed. This report, the Final Opera-
tional Error/Deviation Report (FAA 7210.3), con-
tains detailed information about each error obtained
during the investigation process.

The responsibility for the occurrence of an OE,
except in cases of equipment outage or deficient
procedures, ultimately rests with the air traffic con-
troller who must detect and resolve potential conflicts
before they become errors. However, in many error
analyses, such problems are blamed on some type of
human failure without a deeper investigation of other
contributing factors. It has been the role of the human
factors discipline to address how computer-human
interface (CHI) characteristics can create impedi-
ments to task performance, resulting in errors. This
project extends this approach to consider characteris-
tics of the task domain or environment that could
increase the chances of error.

The discussion of airspace and traffic characteris-
tics in this paper is based on the concept of ATC
complexity (Mogford, Guttman, Morrow, &
Kopardekar, 1995). This construct incorporates the
physical aspects of a sector, such as size or airway
configuration, and factors relating to the movement
of air traffic through the airspace, such as number of
climbing and descending flights. Past research has
identified specific airspace and traffic factors that
contribute to ATC complexity (Mogford, Murphy,
Yastrop, Guttman, & Roske-Hofstrand, 1993).


When using the resulting equation to predict group
membership, there was an overall 58% success rate.
Notably, the individual success rates for predicting
whether a sector would have low or high OE rates were
68% and 63%, respectively. The formula correctly
predicted that a sector would have zero errors for 40%
of the cases. The rates of correctly identifying sectors
from each group by chance would be 33%, 48%, and
18%, respectively, for the no, low, and high-error groups.

4.5.4 Situation Awareness and Operational
Errors

For each of the 85 OEs in the database from Atlanta
ARTCC, a QA Specialist had made a rating of the
presence or absence of SA. Analyses of the OE and
sector data were made to investigate whether control-
ler SA was related to any characteristics of errors or the
sectors in which they occurred.

As in Section 4.1.3, Spearman correlations were
computed between relevant, OE-related variables from
Section 4.2 to screen them for a MANOVA with SA
category as the independent variable. The only signifi-
cant correlation with SA category was for horizontal
separation, 7, = .30, p = .005. A #test for horizontal
separation as a function of SA category showed sig-
nificantly lower separation for errors in the no-SA
group, #(54.69) = -3.06, p = .003 (M = 3.39 versus M
= 3.97). A chi square test showed no significant
difference between no-SA and SA groups as a function
of sector type (low, high, or ultra high).

Further analyses considered the sector characteris-
tics associated with each error. Therefore, the com-
bined set of sector and OE data was used. The statistical
test that addressed sector characteristics as a function
of SA had to take into account that more than one OE
occurred in many sectors. So if several errors were
found in a given sector, that sector’s characteristics
were weighted more heavily in the analyses than with
sectors where there were few errors.

Spearman correlations between sector characteris-
tics in Section 4.1 and SA group (conducted for
screening purposes) revealed that only military traffic
and military operating areas had low correlations, r,=
-.19, p = .085 and r, = -.19, p = .082, respectively. ¢-
tests were conducted and significant differences were

found (at p < .1) for military traffic, (83) = 1.78, p =

.079 (no SA, M = 2.98 and SA, M = 2.39) and for
military operating areas, ¢(83) = 1.70, p = .093 (no SA,
M = 2.94 and SA, M = 2.13). This indicates that there
were few outstanding contrasts in sector measures as
a function of SA.

However, when OE frequency with and without
SA was plotted against overall sector errors, a trend
was evident, as shown in Figure 18. (Sector names and
OE totals are shown on the x-axis.) In this analysis,
sectors with one error were omitted. (Controllers were
always aware in these cases.) It appears that, as the
number of errors in sectors increased, there were more
errors of which the controller had no SA.

The correlation of overall error count with the
number of no-SA errors was r, = 83, p = .000 and with
SA errors was r, = .33, p = .130. For the high SA sectors,
the correlation of overall OE count with non-SA
errors was r, = .86, p = .006 and with SA errors was
r, = 36, p = .386. These results confirm the trend
evident in Figure 18 that error-prone sectors have
many errors where there is no awareness of the devel-
oping problem. Or, as overall OE rate increases, no-
SA error rate rises faster than SA error rate.

Referring to Table 10, there were a number of
statistically significant and near significant differ-
ences between low and high OE sectors. This suggests
that some sector characteristics may negatively affect
SA, which in turn leads to higher OE rates.

4.6 Summary and Discussion

The preceding analysis of OE data will be summa-
rized in this section. The results will then be examined
in terms of questions and predictions identified in the
literature review.

4.6.1 Current Findings

There was an average of two errors per sector at the
Atlanta ARTCC during the period (June 1992 to June
1995), with one sector (Burne) having nine errors. The
45 sectors in the facility varied widely in volume, with a
median of 11,314 cu mi. Twenty-three sectors were low
(0 to FL 230), 13 were high (FL240 to FL340), and 9
were ultra high (FL350 and above). Average traffic
density (based on average traffic volume in each sector
over a three-day period in the 1995 annual facility
review), was 6.1 aircraft per sector.


DOT/FAA/AM-98/14

Office of Aviation Medicine
Washington, D.C. 20591

The Relationship of
Sector Characteristics
to Operational Errors

Mark D. Rodgers

Civil Aeromedical Institute
Federal Aviation Administration
Oklahoma City, Oklahoma 73125

Richard H. Mogford

William J. Hughes Technical Center
Atlantic City, New Jersey 08405

Leslye S. Mogford

Rigel Associates
Marmora, New Jersey 08223

May 1998

Final Report

This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.

Q

U.S. Department
of Transportation

Federal Aviation
Administration


TABLE 10. ANOVA TESTS FOR DIFFERENCES BETWEEN

SECTOR CHARACTERISTICS
OE Group (mean)
Variable Zero Low High F Pp
Amount of climbing/descending traffic 5.3 5.6 6.6 2.16 128
Degree of aircraft mix 5.5 4.0 3.8 2.91 -066
Number of required procedures 45 5.1 6.0 2.78 .074
Weather 47 5.0 6.1 3.75 .032
Radio frequency congestion 41 5.2 6.1 6.20 .004
Average traffic volume 47 5.5 6.0 2.59 .087
Total 16CF score (sum) 70.3 75.7 83.0 3.13 054
Average complexity 269.3 343.8 465.0 5.45 .008

TABLE 11. CORRELATIONS OF OE CHARACTERISTICS WITH NUMBER
OF SECTOR OES AND OE FREQUENCY GROUP

Correlations

Variable Number of OEs OE Group
(low & high only)
Number of aircraft in the sector r=.19, p= .084 ft, =.20, p= .067
Complexity r=.28, p=.011 If, =.25, p= .021
Workload index r=.24, p= .027 re =.21, p= .052
Vertical separation r= .20, p=.073 NS
Position combined N/A fs = -.29, p= .007

TABLE 12. REGRESSION ANALYSIS OF OE DATA

REGRESSION STATISTICS

MULTIPLE R 58
MULTIPLE A? 34
ADJUSTED FR? 31
SE OF ESTIMATE 1.73
ANALYSIS OF VARIANCE
ss at MS FRATIO
REGRESSION 64.77 2 32.39 10.82
RESIDUAL 125.67 42 2.99 p=.000
VARIABLES IN THE EQUATION
VARIABLE b SE b t p(2 TAIL)
FREQ. CONG. 76 A7 56 4.37 .000
REST. AREAS 34 14 32 2.47 017
CONSTANT 64.59

40



Couluris and Schmidt (1973) noted that sector
design can affect controller workload and, presum-
ably, the potential for OE occurrence. Number of
handoffs, coordination, pointouts, and structuring
and bookkeeping events:

.fesult from, or are influenced by, the existence and
design (shape) of the sectors. The additional work
created can be thought of as the cost of sectorization.
Although they are still related to traffic and route
parameters, they can be varied. For example, a sector
boundary that crosses a highly traveled route will create
a larger work load (from the above four work-producing
event types) than a boundary across a sparsely traveled
route. (p. 657)

This research suggests that sector structure, in
terms of boundary location and shape, airway con-
figuration, and intersections, can affect conflict fre-
quency. Such factors as traffic flow rate, average
velocity, separation requirements, number of flight
levels, and transitioning traffic can also influence
outcomes.

The following studies sought to classify air traffic
controller errors and may shed some light on possible
sector-related causes of OEs. They were screened for
possible factors that could help orient the SATORI-
based research on Atlanta ARTCC airspace.

Empson (1987) applied a human error classifica-
tion system to military air traffic controllers in the
United Kingdom over a two-year period. The subjects
performed two distinct roles, that of radar director
and radar approach. The radar director controller
gave heading and altitude instructions to aircraft, kept
them separated, and sequenced them for approaches
to the airfield. The radar approach controller was
responsible for aircraft within a 30-mile radius of the
field and initially identified aircraft inbound for land-
ing. This person also handled aircraft departing from
the field. The radar director was responsible for
flights nearing the field while the approach con-
trollers handled aircraft within the approach area.
Approach controllers handled about twice as much
traffic as the directors.

Eight controllers (four in each category) were ob-
served over two, 24-hour periods. A total of 131 errors
were observed and categorized as discrimination errors,

program-based (or action-related) errors, and errors
relating to memory functions. Errors made by direc-
tors were higher than for approach controllers. This
was in spite of the fact that directors had aircraft on
frequency 36 percent of the time compared with 79
percent for approach controllers. The error rate for both
controller types increased with traffic load.

The author suggested the reason for these results
might be that the director’s job was more difficult.
The director typically had to accept fast reconnais-
sance jets that were low on fuel and sequence them for
handoff for landing. Handoffs had to occur at a
precise heading and altitude. The approach control-
ler, on the other hand, experienced much less time
pressure and dealt with aircraft with various destina-
tions. In a related paper by Langan-Fox and Empson
(1985), it was suggested that another reason for the
higher workload in the director position could be that
the director tasks were force-paced. Thatis, in the case
of the approach controller, the presentation rate of
events were system controlled as opposed to worker
controlled (self-paced). Other research (Bertelson,
Boone, and Renkin, 1965) demonstrated almost
error-free performance for workers in a self-paced
work setting but showed a dramatic increase in
error rates when the self-pace rates were imposed
upon the operators.

This research, apart from being a useful application
of a human error taxonomy, demonstrates possible
effects of airspace structure, procedural demands, task
characteristics, and traffic type on controller errors.
The findings support the assumption that an analysis
of such features can lead to a better understanding of
controller error.

In contrast to these observations of working con-
trollers, the following papers focused on the analysis
of OE databases.

Kinney, Spahn, and Amato (1977) conducted a
study of system errors as recorded in the US System
Effectiveness Information System (SEIS) for the years
1974, 1975, and 1976. The data for en route centers
showed that, of the 564 errors recorded for the 3 years,
95% of direct causes were attributed to attention,
judgment, or communications. The same categories
accounted for 71% of contributing causes. Other
contributing factors included stress (0.2%), equipment


The emphasis of the literature review was on air-
space- and traffic-related factors that contribute to
OE occurrence. Although the papers discussed con-
tained information on other factors that affect error
frequency, they were not the focus of this effort. As
much as possible, the contents of each reference were
screened for insights on how sector design, traffic flow
patterns, procedures (such as those found in Letters of
Agreement [LOAs]), and other airspace factors cause,
relate to, or affect OE incidence. Other research
findings or hypotheses were only reported as needed
to support the discussion of this theme.

3.2 Summary of the Literature

Several of the reviewed papers discussed theoreti-
cally-derived airspace issues that could be related to
OEs. For example, Arad (1964a) conducted an ana-
lytical and empirical study of workload in relation to
sector design. He divided controller workload into
three categories: the background load involved in
working the position, independent of aircraft activity;
the routine load of controlling a “standard” aircraft,
irrespective of its interactions with other aircraft; and
the airspace load imposed by the natural tendency of
uncontrolled traffic, in a free-flow environment, to
converge in unsafe ways, thus requiring control actions.

Arad (1964a) developed a mathematical expression
that he suggested would account for the number of
aircraft conflicts. The variables in the equation in-
cluded rules of separation, average traffic speed, num-
ber of aircraft under control, sector size, and flow
organization. The last term was not clearly defined,
but was described as “a number that quantifies the
flow organization and numerically relates the vari-
ables... to the actual numerical value of the conflict
rate...” (p. 29).

A report by Arad (1964b) described further work
on the analysis of the above load factors and suggested
that sector design could greatly affect the routine load
imposed on the controller. For example, if traffic flow
tends to be north/south, then establishing sector
boundaries parallel with flow imposes less work on
controllers than if they are established east/west, or

perpendicular to the traffic flow. Therefore, a
rectangular sector with its long side parallel with the
direction of traffic flow is most efficient if traffic tends
to flow in one direction. If traffic flow is more random,
a circular-shaped sector is more effective and will pre-
sumably result in fewer OEs.

Along similar lines to these papers, Siddiqee (1973)
attempted to develop a mathematical model for pre-
dicting the expected duration of aircraft conflicts at
air route intersections. The equations are not relevant
for this review, but one of the author’s assumptions is
noteworthy:

«in the en route environment, aircraft fly essentially
level at certain standard altitudes. The standard altitude
increments used are large enough to ensure adequate
vertical separation, with allowance for flight technical,
altimeter, and pilot errors. Thus, in en route environ-
ments, conflict situations arise mainly because of loss of
the horizontal separation among aircraft flying at the
same altitudes. (p. 59)

Schmidt (1976) described a sector workload model
intended to aid in the design and evaluation of air-
space. The author defined ATC workload as “the
frequency of occurrence of events which require deci-
sions to be made and actions to be taken by the
controller team, and the time required to accomplish
the tasks associated with these events” (p. 531). Event
categories included potential conflicts between aircraft
at air route intersections, potential aircraft-overtaking
conflicts along air routes, and routine procedural events.

Schmidt (1976) noted that the expected frequency
and duration of crossing and overtaking conflicts can
be predicted by traffic flow rate, aircraft separation
standards, route geometry, and aircraft velocity. Con-
flicts at an air route intersection are related to aircraft
flow rate, velocity along each route, minimum aircraft
separation requirements, angle of intersection be-
tween the routes, and number of flight levels. The
author developed equations that purportedly would
predict conflict event frequency and added that the
amount of transitioning traffic had to be factored into
the calculation.


AVG_COM_

AVG_DEN_

CLDC

COOR

CUVOL

FPAPCT

FREQ

FUNC

INTR

JET

MIX

(
P=

mn
W

(Coefficient /

0332
45)
+828

+2633

45)
-081

4821

45)
-001

+0702

45)
-647

-3913

45)
-008

+2453

45)

= .104
+4366

45)
-003

+0880

45)

= .565

-0710

45)
643

+1456

45)
+340

-5476

45)
-000

NOINTSX

(
P=

Correlation Coefficients

.0947
45)
-536

0768
45)
-616

+2984
45)
-046

~2602
45)
-084

+2963
45)
-048

+2063
45)
.174

+0101
45)
-947

+2497
45)
-098

.0721
45)
-638

+0292
45)
-849

-4776
45)
-001

OTHRTS

P=

0839
45)
-584

+4267

45)
.003

+4620

45)
-001

-1087

45)
477

+7272

45)
-000

.1144

45)
454

-1693

45)
+266

- 0369

45)
-810

-0791

45)
-606

+2528

45)
+094

8466

45)
+000

PROC

(
P=

oA
W

oA
aw

A
iy

ga
W

ga
W

P=

(Cases) / 2-tailed Significance)

+5239
45)
-000

.1773
45)
+244

+6650
45)
+000

+5227
45)
-000

+2464
45)
103

-1099
45)
+472

-5505
45)
-000

- 8828
45)
+000

+3623
45)
+014

-5121
45)
-000

+2100
45)
-166

“. " is printed if a coefficient cannot be computed

A7

RAD

+2989
45)
+046

+2839
45)

= .059

-0556
45)
.717

.0797
45)

= .603

-0588
45)
-701

.1347
45)
.378

2495
45)
-098

0175
45)
+909

+2669
45)
.076

.1016
45)
-507

0277
45)
-857

REST

-0476
( 45)
P= .756

-0267
45)
+862

oA
W

+2230
( 45)
P= .141

+0021
45)
-989

oA
Wl

-.2451
45)
.105

oA
w

+2471
( 45)
P= .102

-.2176
45)
2151

gn
w

+1059
45)
+489

yA
W

-.0110
45)
P= .943

-0000
( 45)
P=1.000

-.2146
( 45)
P= .157


TABLE 4. SUMMARY OF LITERATURE REVIEW

Source

Factors or Issues

Arad (1964)

Conflicts related to rules of separation, average traffic speed,
number of aircraft under control, sector size, and flow
organization.

Arad, et al. (1964)

Routine load on controller affected by placement of sector
boundaries with respect to traffic flow.

Siddiqee (1973) Conflicts in en route airspace occur due to a loss of horizontal
separation between aircraft at the same altitudes.
Schmidt (1976) Conflicts predicted by traffic flow rate, separation standards, route

geometry, aircraft speed, aircraft flow rate, angle of airway
intersection, number of flight levels, and amount of transitioning
traffic.

Couluris and Schmidt (1973)

Cost of sectorization is additional workload (coordination) imposed
by placement of sector boundaries.

Empson (1987)
Langan-Fox and Empson (1985)

Controller workload is related to airspace structure, procedural
demands, traffic type, and control over task presentation rate.

Kinney, et al. (1977)

OEs occur under low to moderate workload and moderate
complexity. In en route centers, 95% of errors attributed to
attention, judgment, or communications. Most errors occur in level
flight.

Schroeder (1982)

Most errors occur under light or moderate workload. Other
aspects of the situation [sector factors?], apart from traffic volume,
determine workload. Coordination is a direct or contributing factor
in many errors.

Stager and Hameluck (1990)
Stager, et al. (1989)

Definitions of direct and contributing causes. OEs occur under low
to moderate workload conditions. Causes are attention, judgment,
and communication problems.

Redding (1992)

Failure to maintain SA causes most errors under moderate traffic
load. Communication, coordination, and misuse of radar data
account for most errors.

Schroeder and Nye (1993)

OEs occur under average or lower traffic complexity. Problems
with radar display, communication, coordination, and data posting
most frequent causes.

Rodgers and Nye (1993)

Most OEs occur with one aircraft in level flight and another
descending or ascending. Most moderate errors are between
aircraft in level flight. Horizontal, not vertical, separation varies
with severity. Higher horizontal separation for SA OEs.

Fowler (1980)

Sector complexity effected by problems with coordination,
procedures, LOAs, and weather.

Buckley, et al. (1983)

Sector geometry and traffic density interact to affect controller
performance.

Stein (1985)

Controller workload is related to clustering of aircraft in a small
amount of airspace, number of hand-offs outbound/inbound, and
total number of flights handled.

36





TABLE 13. RELATIONSHIP OF FINDINGS TO LITERATURE REVIEW

Source Factors or Issues Relevant Factors

Arad (1964) Conflicts related to rules of separation, average Correlations with OE rate:
traffic speed, number of aircraft under control, Traffic volume higher,
sector size, and flow organization. Complex routings higher,

Sectors smaller in high OE sectors.

Arad, et al. Routine load on controller affected by placement | No relevant findings

(1964) of sector boundaries with respect to traffic flow.

Siddiqee (1973) | Conflicts in en route airspace occur due toaloss_ | SA errors had more horizontal
of horizontal separation between aircraft at the separation than no-SA errors.
same altitudes.

Schmidt (1976) Conflicts predicted by traffic flow rate, separation | Correlations with OE rate:
standards, route geometry, aircraft speed, aircraft | Traffic volume,
flow rate, angle of airway intersection, number of {| Complex routings,
flight levels, and amount of transitioning traffic. Climbing/descending traffic.

Couluris and Cost of sectorization is additional workload Correlations with OE rate:

Schmidt (1973) (coordination) imposed by placement of sector (Coordination NS).
boundaries. Coordination problems cited in

11% of OEs.

Empson (1987) Controller workload is related to airspace Correlations with OE rate:

Langan-Fox and | structure, procedural demands, traffic type, and Complex routings,

Empson (1985) control over task presentation rate. Required procedures,

Aircraft mix.

Kinney, et al. OEs occur under low to moderate workload and OEs occur with above average

(1977) moderate complexity. In en route centers, 95% of | traffic load. Most OEs rated as
errors attributed to attention, judgment, or moderately complex.
communications. Most errors occur in level flight.

Schroeder (1982) | Most errors occur under light or moderate OEs occur with above average
workload. Other aspects of the situation [sector traffic load. Most OEs rated as
factors?] apart from traffic volume determine moderately complex. Coordination
workload. Coordination a direct or contributing not correlated with error count but
factor in many errors. cited as cause in 11% of errors.

Stager and Definitions of direct and contributing causes. OEs | OEs occur with above average

Hameluck occur under low to moderate workload. Causes traffic load. Most OEs rated as

(1990), Stager, et | are attention, judgment, and communication moderately complex.

al. (1989) problems.

Redding (1992) Failure to maintain SA cause of most errors under | High error sectors also had many
moderate traffic load. Communication, no-SA errors.
coordination, and misuse of radar data account for | Primary causes of OEs: 29%
most errors. communication; 81% problems

with radar display.

Schroeder and OEs occur under average or lower traffic OEs occur with above average

Nye (1993) complexity. Problems with radar display, traffic load. Most OEs rated as

communication, coordination, and data posting
most frequent causes.

moderately complex. Primary
causes of OEs: 29%
communication; 81% problems
with radar display

Rodgers and Nye
(1993)

Most OEs occur with one aircraft in level flight and
another descending or ascending. Most moderate
errors are between aircraft in level flight.
Horizontal, not vertical, separation varies with
severity. Higher horizontal separation for SA OEs.

SA errors had more horizontal
separation than no-SA errors.

41



and add variables to OpenCreate. Based on the litera-
ture review, some candidate measures might be: a
count of airway crossings of sector boundaries (to
evaluate coordination requirements), aircraft speeds
at the time of the OE, route complexity, a count of
event- versus controller-driven activity (Langan-Fox
and Empson, 1985), and flight stage (climbing, descend-
ing, or level) at the time of the OE. The characteristics
inherent in the dimensions of the factor analysis could
also be melded into more global questions.

Collection of additional ratings of ATC complex-
ity from Atlanta Center might help stabilize the 16CF
data. (Only one controller from each area was used in
this study.) Given that previous research found that
interrater reliability between controllers from the same
areas tended to be only moderate, further steps should
be taken to ensure clear definition of each factor,
perhaps using graphics combined with written de-
scriptions. A computerized version of the 16CF ques-
tions may be worth exploring.

To summarize, this research has shown that high-
OE sectors are characterized by problematic weather,
radio frequency congestion, high total 16CF com-
plexity, high annual review average complexity, and
small size. There is also evidence that these sectors
tend to have more climbing/descending traffic, a
uniform aircraft mix, frequent required procedures,
and higher traffic volume. The general dimensions
that describe sector and traffic characteristics are
traffic activity, size, and military. Finally, there is
limited evidence that OE probability can be predicted
using a subset of these variables.

The separate analysis of controller situational aware-
ness (SA) during the development of OEs is important
in that, in these data, 73% of the controllers were not
aware of the developing error. Presumably, awareness
would have prevented many of these errors from
occurring, as suggested by Redding (1992). The only
sector or traffic characteristic that was clearly differ-
ent between not aware and aware OE groups was
horizontal separation, as found by Rodgers and Nye
(1993). However, this result may have been due to the
fact that, without awareness, the error was more fully
advanced before controller intervention occurred.

23

It was found that high-error sectors tended to have
more no-SA errors. It may be that the presence of
awareness of a developing error is a mediating factor
controlling the frequency and severity of errors in a
given sector. If sector or traffic characteristics tend to
somehow interfere with general controller SA, it can
be expected that more errors will occur, and they will
often be rated as no-SA OEs. Thus, higher ATC
complexity may result in the kind of high cognitive
loading that contributes to a reduction in SA and leads
to an elevated probability of error.

Although direct comparisons between the current
findings and previous research in this area were not
always possible, there were many links. It was also
notable that there was a 24% rise in the attribution of
OE causation to problems with the radar display.
Coordination and data posting were cited less fre-
quently, and communication remained at about 30%.
This represents significant increase in problems with
misreading or misusing visually-displayed data with
the concomitant negative impact on SA. A more
detailed analysis of this finding is indicated. Changes
to information display methods may be required to
eliminate some of the causes of these errors.

This project has resulted in a review of the sector-
and traffic-related factors associated with OEs. Armed
with OE data from the Atlanta ARTCC from a variety
of sources (including SATORI), it has been possible
to explore relationships between a range of ATC
complexity measures and errors. Although this study
must be considered as exploratory, it has succeeding
in demonstrating a role for sector complexity, as
measured by an array of variables, in OE incidence. It
has also confirmed the Rodgers and Nye (1993) asser-
tion that SA may be an important mediating factor in
the creation and eventual severity of errors. With a
larger data set and refined measurement techniques, it
may be possible to generate reliable, statistically sound
tules for the prediction of OEs, and useful guidelines
for sector reconfiguration and design. This could
provide direct benefits for the development of proce-
dures and automation tools for reducing errors and
enhancing safety in the NAS.


MIL

MIX

NOINTSX

OTHRTS

PROC

REST

RTNG

SHELF

SIZE

SQ_SP

(Coefficient / (Cases)

VICTOR

(
P=

0951
45)
-534

-6223

45)
-000

4634

45)

= .001
9054

45)
+000

-0583

45)
704

~1542

45)
+312

+3226

45)
-031

-1317

45)
+389

+4125

45)

= .005
+6531

45)
-000

.1144

45)

= 454

Correlation Coefficients

VOL

(

1795
45)
+238

.1948

45)

= .200
-0138

45)
+928

+2226

45)

= .142

5653

45)
-000

.2156

45)
155

.1715

45)
260

-3808

45)

= .010
-1708

45)
+262

+2043

45)
-178

+5330

45)
-000

VORTAC

(
P=

oN
W

on
W

+2835
45)
+059

+1241
45)

= .417

.3588
45)
-015

+2923
45)
-051

-0470
45)
+759

+2068
45)
-173

+2130
45)
-160

+0899
45)
-557

-1623
45)
-287

.1994
45)
189

-0718
45)
-639

WX

/ 2-tailed Significance)

Al4

+2020
45)
.183

-1692
45)
+266

-1482
45)
+331

.1434
45)

= .347

-5197
45)
-000

-0904
45)

= .555

-1058
45)
-489

«3781
45)
+010

.0182
45)
+905

-0843
45)
+582

-3606
45)
015

is printed if a coefficient cannot be computed


sector error count increased, there was a greater pro-
portion of errors were the controller did not have SA.
Thus, high-error rate sectors were characterized by
low SA for error development.

The results of the data analysis were reviewed for
relationships to the literature survey. Of note was the
24% increase in error causation assigned to problems
with the radar display. Previous research found that
most errors occurred in low to moderate workload.
The current data set showed that errors were reported
with an average of eight aircraft in the sector and with
moderate complexity. This density level was signifi-
cantly higher than the base rate of 6.5 aircraft. It was
not possible to explore the issue of coordination
problems extensively. There was some indication that
sectors with climbing and descending traffic experienced
more OEs, thus corroborating some previous findings.

vi

It was concluded that further data on normal At-
lanta ARTCC traffic flows are needed to determine if
OE frequencies departed from expected proportions.
The factor analysis produced useful information about
the underlying dimensions (Traffic Activity, Mili-
tary, and Size) characterizing sectors. One facility
review variable, Average Density, was not related to
other traffic load measures. This warrants further
study. Several results demonstrated a relationship
between sector complexity and OE rate. Such find-
ings, when validated, could assist with traffic manage-
ment and sector design activities.

Given the importance of situation awareness for
avoiding operational errors, evidence was found that
increased sector complexity may be associated with
reduced situation awareness and may lead to a larger
number of, and more severe, errors.


Secondary causal factors were largely attributed to
communication errors and radar display problems.
Table 8 shows a summary of all reported causal
factors.°

4.5.2 Operational Error Conditions

This section reviews the quantitative OE data,
analyzing the circumstances under which errors oc-
curred. Figure 6 shows the distribution of OEs by
time of day. It appears that most errors occurred
between 0800 and 2000 hours. This probably corre-
sponds with normal traffic flows at the Atlanta
ARTCC, although no data were available on overall
traffic count over time.

Figure 7 is a distribution of OEs by flight level.
There were spikes in the OE count between 15,000 ft
and 20,000 ft and again between FL300 and FL350.
Figure 8 is another rendition of OE by level, except
that it shows the distribution of errors by sector type
(low, high, and ultra high).

Figure 9 is a combination of Figures 4 and 8,
showing the proportion of sectors and OEs by sector
type. It appears that a disproportionately high number
of errors occurred in high sectors while fewer than
might be expected were recorded in ultra high sectors.
Acchi square test of these data showed that there were
significantly different proportions of errors than would
be expected given the number of sectors in each group,
X7(85)=12.43, p = .002. However, this result might be
related to normal traffic flow patterns in each sector type.

Traffic count data at the time of each error are
plotted in Figure 10, The distribution of errors by
number of aircraft in the sector when separation was
lost approximates a normal curve with a mean of 8.0
aircraft and a standard deviation of 2.9 aircraft. When
comparing these data with the average density of
sector traffic reported in Figure 5, it is evident that
traffic density at the time of an error was, on the
average, 1.5 aircraft higher.® This difference was sig-
nificant with 2(134.94) = 4.12, p = .000.

Figure 11 is a plot of OE count by estimated
complexity rating (from the OE report). The com-
plexity of the air traffic situation at the time of the OE
was assigned after the error occurred on a scale of 1
(low) to 5 (high). Average complexity was 3.4 with a
standard deviation of 1.2 and a median of 4.0. Most
errors were found in the moderate (3 or 4) range. OE
report complexity had low, but significant, correla-
tions with average (annual review) sector complexity
(r= .24, p = .030) and total 16CF ATC complexity (r
= 40, p= .000). It had a high correlation with number
of aircraft in the sector at the time of the error (r= .86,
Pp = .000) and low, but significant, correlations with
16CF traffic volume (r = .24, p = .029) and average
(annual review) sector traffic density (r = .21, p = .052).

Figure 12 is a plot of OE frequency by workload
index. Workload was calculated based on an approach
developed by Rodgers and Nye (1993). The index is
the sum of the z scores for complexity and traffic
count at the time of the error. It takes into account
both sector and traffic factors in estimating controller
workload. As can be seen in Figure 12, the distribu-
tion is somewhat negatively skewed, indicating that
workload for many errors was higher than the average for
all errors.

Figure 13 is a histogram of OEs by amount of
vertical separation. The errors were divided into two
groups: below FL290 and FL290 and above, given
that there are two separation standards in en route
airspace. (Below FL290, 1000 ft is required, while at
FL290 and above, 2000 ft is the minimum.) As seen
in the graph, two concentrations of separation at
closest point of approach were found, corresponding
to the two standards. Although there were some errors
with less than 1000 ft of separation at the higher
levels, most had at least 1000 ft.

Figure 14 depicts the horizontal separation be-
tween aircraft pairs at the time of OEs. The distribu-
tion is negatively skewed with a mean of 3.6 mi, a
standard deviation of 0.9 mi, and a median of 3.8 mi.
Most errors occurred with 3 or more miles of horizon-
tal separation remaining.

5 The percentage total sums to more than 100 because more than one cause may have been attributed to each error.

6 The mean of the annual review average density for all sectors was 6.5 aircraft in this calculation. The average traffic density for each sector
associated with an error was included, resulting in some sectors being counted more than once in the average. This resulted in a different mean

than that computed for Figure 5.


MIL

MIX

NOINTSX

OTHRTS

PROC

REST

RTNG

SHELF

SIZE

SQ_SP

FREQ

(
P=

P=

-0269
45)
-861

- 4366
45)
-003

+0101
45)
947

-1693
45)
-266

-5505
45)
-000

+2495
45)
+098

+2176
45)
+151

-3763
45)
-011

+2077
45)
2171

1444
45)
+344

~4325
45)
-003

Correlation Coefficients

FUNC

(
P=

+2592
45)
+086

0880
45)
-565

~2497
45)

= .098

-0369
45)
-810

- 8828
45)
-000

0175
45)
+909

-1059
45)
+489

+5365
45)

= .000

0914
45)
-550

.1416
45)

= .353

- 4313
45)
-003

HUB

2476
45)
-101

+0710

45)
643

-0721

45)
-638

-0791

45)
-606

-3623

45)
-014

+2669

45)
.076

+0110

45)

= 943
-2158

45)
2155

-0376

45)
-806

-1047

45)
+494

+4593

45)
+002

INTR

(
P=

(Coefficient / (Cases) / 2-tailed Significance)

.1158
45)
+449

-1456
45)

= .340

+0292
45)
-849

+2528
45)
-094

-5121
45)
-000

1016
45)

= .507

-0000
45)
1.000

+3050
45)
-042

1514
45)
+321

0096
45)
+950

+3193
45)
-033

"|, "is printed if a coefficient cannot be computed

AS

JET

0992
45)
.517

5476

45)
-000

4776

45)

= .001
- 8466

45)
-000

+2100

45)
-166

+0277

45)
-857

-2146

45)
157

0112

45)
+942

-3105

45)
-038

6547

45)
-000

-1510

45)
.322

MIL

1.

0000
45)

0445

45)
+772

-1858

45)
+222

.1221

45)
+424

1406

45)
+357

-4327

45)
+003

+5929

45)

= .000

+1385

45)
+364

+2802

45)
+062

-0560

45)
715

-1842

45)
+226


- - Correlation Coefficients - -

RING SHELF SIZE SQ_SP TOTALOE TOTCOMX
TOTALOE +2363 -.0684 -.1221 +0929 1.0000 +3821
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .118 P= .655 P= .424 P= .544 P=. P= .010
TOTCOMX - 6004 - 0283 -0108 -6180 +3821 1.0000
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .854 P= .944 P= .000 P= .010 P=.
VICTOR -1317 -.4125 -6531 1144 -.0241 -.0382
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .389 P= .005 P= .000 P= .454 P= .875 P= .803
VoL -3808 -.1708 +2043 +5330 -3338 - 7066
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .010 P= .262 P= .178 P= .000 P= .025 P= .000
VORTAC -0899 +1623 .1994 -0718 -.3050 .0157
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .557 P= .287 P= .189 P= .639 P= .042 P= .918
Wx -3781 -0182 +0843 +3606 +4493 6685
( 45) ( 45) ( 45) 45) ( 45) ( 45)

(
P= .010 P= .905 P= .582 P= .015 P= .002 P= .000

(Coefficient / (Cases) / 2-tailed Significance)

" . " is printed if a coefficient cannot be computed

Al2


Number of OEs

@ Below FL290
Above FL290

0 200 400 600 800 1000 «1200 =«:1400— 1600 -~— 1800
Vertical Separation (ft)

FIGURE 13. DISTRIBUTION OF OEs BY VERTICAL SEPARATION

Number of OEs

1 15 2 25 3 35 4 45 5

Horizontal Separation (mi)

FIGURE 14. DISTRIBUTION OF OEs BY HORIZONTAL SEPARATION

31



Number of OEs

Below FL290
& Above FL290|

RMS Separation (ft)

FIGURE 15. MINIMUM RMS SEPARATION FOR OEs

Number of OEs

45

35

1 2 3
Number of People Working

FIGURE 16. NUMBER OF CONTROLLERS WORKING
AT THE TIME OF THE ERROR

32



APPENDIX A:
PEARSON CORRELATION MATRIX OF SECTOR VARIABLES

- - Correlation Coefficients - -

AVG_COM_ AVG_DEN_ CLDC COOR CcuVvOoL FPAPCT
AVG_COM_ 1.0000 +2762 -5744 +2129 ~.2830 .1908
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P=. P= .066 P= .000 P= .160 P= .060 P= .209
AVG_DEN_ +2762 1.0000 -.0895 .1481 +3294 .1156
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .066 P=. P= .559 P= .332 P= .027 P= .449
CLDC .5744 ~.0895 1.0000 +3129 -.6169 +2218
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .559 P=. P= .036 P= .000 P= .143
COOR +2129 -1481 .3129 1.0000 -.1467 -.0196
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .160 P= .332 P= .036 P=. P= .336 P= .898
CUVOL -.2830 ~3294 -.6169 -.1467 1.0000 -0175
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .060 P= .027 P= .000 P= .336 P=. P= .909
FPAPCT -1908 .1156 +2218 -.0196 -0175 1.0000
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .209 P= .449 P= .143 P= .898 P= .909 P=.
FREQ +5430 +2178 +2537 -3168 -.0954 -.0245
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .151 P= .093 P= .034 P= .533 P= .873
FUNC .5162 +2229 - 6369 -5070 -.2088 -1233
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .141 P= .000 P= .000 P= .169 P= .420
HUB -5288 -.0092 -5560 -.0005 -.1656 -1217
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .000 P= .952 P= .000 P= .997 P= .277 P= .426
INTR +3572 +2597 -1920 +3394 +0142 +0207
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)
P= .016 P= .085 P= .206 P= .023 P= .926 P= .892
JET -.2158 +3615 -.5612 -.1441 +7146 -.0935
( 45) 45) ( 45) ( 45) ( 45) ( 45)

(
P= .155 P= .015 P= .000 P= .345 P= .000 P= .541

(Coefficient / (Cases) / 2-tailed Significance)
" , " is printed if a coefficient cannot be computed

Al


The average complexity was 3.4 (out of 5), and most
errors were in the moderate range. When a derived
workload measure was calculated, it appeared that
OEs occurring in high-error sectors had marginally
higher workload.

Although several authors suggested that coordina-
tion could be a factor in OE incidence (Arad, et al.,
1964; Couluris and Schmidt, 1973; Stein, 1985;
Fowler, 1980; Mogford, et al., 1993; Rodgers and
Manning, 1995), there was not much evidence in the
current data set to support this hypothesis. The only
measure of the amount of coordination required in a
sector was one item on the 16CF, and this variable was
not correlated with OE frequency nor did it discrimi-
nate between groups of sectors with different OE
severities, As noted earlier, coordination was men-
tioned as a factor in OE occurrence in 11% of the
reports during the period. However, as a whole, the
data do not add much to our understanding about the
effects of sector boundary placement and coordina-
tion on OE occurrence.

In the literature review, some reports indicated that
conflicts usually occurred between aircraft in level
flight (Siddigee, 1973; Kinney, et al. 1977). Rodgers
and Nye (1993) found that most errors involved one
aircraft in level flight, and one that was climbing or
descending, though most moderately severe errors
involved aircraft in level flight. The OE data reviewed
in this report did not contain information relevant to
this topic. However, there was a low, but significant
correlation between number of OEs and amount of
climbing and descending traffic. This suggests that
transitioning traffic had some role in error generation.

Rodgers and Nye (1993) also found that 30% of
the 1053 errors they analyzed were rated as moderate
in severity while 70% were minor errors. In compari-
son, the current data show 16% moderate and 84%
minor errors, with a much smaller set of 85 errors.

Finally, there is unfortunately not much relation-
ship between the approach used in this research and
the recent work by Rodgers and Manning (1995).
Aircraft density was generally higher at the time of an
OE, but it was not possible to address the other variables
used in this previous study, given the data at hand.

20

5. CONCLUSIONS

Several useful and important findings emerged asa
result of analyzing the 1992 to 1995 operational error
(OE) reports and investigating relationships between
sector and traffic factors and OEs. These will be
discussed in this section and recommendations for
further study will be enumerated.

Using a combination of OpenCreate and other data,
it was possible to generate useful statistics regarding
Atlanta Air Route Traffic Control Center (ARTCC)
sector features. Of note was the wide range in sector size.
These descriptive data may be useful for comparing en
route facilities during future research efforts.

Without normative data on the daily traffic flows
and typical altitudes, itis difficult to determine whether
OE frequency departed from expected proportions, as
determined by normal traffic density patterns. It may
also be that ultra-high sectors are often combined
with high sectors. However, it appears that more OEs
occurred in high sectors, and fewer in ultra-high
sectors, than would be anticipated, based on sector
counts. Future research should gather facility baseline
data to support such comparisons.

It is evident that errors tended to occur at above
1995 average traffic density. However, it is again
difficult to know how much busier these sectors were
relative to normal ranges of traffic density at the time
of OEs. As found by Kinney, et al. (1977) and others,
the traffic noted in the OE reports may fall within the
moderate range. Further data from the AtlantaARTCC
would help determine whether OE frequency de-
parted from expected proportions, as determined by
normal traffic density patterns.

OE complexity rating (from the OE report) had a
high correlation with aircraft density in the sector at
the time of the OE. It had much lower correlations
with other general sector volume and complexity
ratings. This suggests that more immediate factors,
such as traffic density during the OE period, are
important in this rating, as compared to overall sector
characteristics.

Of interest was the moderate correlation of error
severity and sector type (L, H, or UH), indicating that
sectors under FL230 tend to have more severe errors.


AVG_COM_

AVG_DEN_

CLDC

COOR

CUVOL

FPAPCT

FREQ

FUNC

INTR

JET

(Coefficient /

+5291

45)
-000

+4819

45)
-001

+3704

45)

= .012

+0315

45)
-837

Correlation Coefficients

FUNC

(
P=

5162
45)
-000

+2229

45)
.141

+6369

45)

= .000

+5070

45)
-000

+2088

45)
.169

-1233

45)
+420

5291

45)
-000

-0000

45)

+2319

45)
-125

-5898

45)
-000

+2477

45)

= .101

HUB

-5288

45)
-000

.0092

45)

= .952
+5560

45)
-000

-0005

45)

= .997

1656

45)
+277

.1217

45)
+426

-4819

45)
-001

+2319

45)
+125

-0000
45)

0784

45)
609

.0944

45)
537

INTR

(
P=

(Cases) / 2-tailed Significance)

3572
45)
.016

.2597

45)
.085

+1920

45)

= .206

.3394

45)
+023

.0142

45)
+926

-0207

45)
+892

3704

45)

= .012
+5898

45)
-000

.0784

45)

= .609
- 0000

45)

-1128

45)

= .461

" . "is printed if a coefficient cannot be computed

A4

es

MIL

-.0172

45)
-911

0710

45)
+643

+0260

45)
.865

+2505

45)

= .097

-1182

45)
+439

.1326

45)
+385

.0269

45)
-861

+2592

45)
-086

+2476

45)

= .101

.1158

45)
449

-0992

45)
.517


AVG_COM_

AVG_DEN_

CLDC

COOR

CuUVOL

FPAPCT

FREQ

FUNC

HUB

INTR

JET

VICTOR

P=

1124
45)
462

+4612

45)
001

-5220

45)
-000

-0198

45)
-898

7778

45)

= .000
-1142

45)
2455

+1622

45)
287

0895

45)
+559

0551

45)
«719

+2845

45)
+058

+8879

45)
-000

Correlation Coefficients

VoL

P=

-5943
45)
-000

+2614
45)
-083

-3938
45)
-007

-1560
45)

= .306

- 0806
45)
-599

-0023
45)
+988

-6815
45)
+000

.5107
45)

= .000

-6355
45)
-000

- 3882
45)
+008

.1451
45)
+342

VORTAC

(
P=

(
P=

mwa
w

on mA
w W

ga
W

- 0307
45)
-842

+3663
45)
-013

+0800
45)
+602

-0011
45)
+994

.2872
45)
-056

0075
45)
-961

.1057
45)
-490

-0149
45)
+923

.0878
45)
-566

+2792
45)
-063

+2162
45)
154

Wx

(Coefficient / (Cases) / 2-tailed Significance)

+5641
45)
+000

.1766
45)
+246

~4149
45)
-005

-3467
45)
+020

+2065
45)
.174

-1607
45)
+292

+6244
45)
.000

+4854
45)
001

-3879
45)

= .008

+3248
45)
+030

-0322
45)

= .834

“, “is printed if a coefficient cannot be computed

Al3




TABLE 7. SECONDARY CAUSAL FACTORS

Secondary Causal Factor Number | Percent
Communication error 10 42
Radar display: Inappropriate use of displayed data 8 33
Coordination 5 21
Radar display: Misidentification: Climbed aircraft with similar call sign 1 4

TABLE 8. SUMMARY OF ALL CAUSAL FACTORS

Summary of Causal Factors | Number | Percent
Radar display 69 81
Communication error 25 29
Coordination 9 11
Data posting 5 6
Unknown 1 1

TABLE 9. CORRELATIONS OF SECTOR VARIABLES WITH NUMBER
OF SECTOR OEs AND OE GROUP

Correlations
Variable Number of OEs OE Group
Number of major airports N/A = -.34, p= .021
Number of VORTACs r=-.31, p= .042 | r.=-.25, p= .094
Amount of climbing/descending traffic r=.33,p=.029 | r,=.29, p=.051
Degree of aircraft mix r=-.28, p=.060 | r,=-.33, p= .025
Number of multiple functions r= .33, p= .028 Tg = .26, p= .084
Number of required procedures r= .38, p=.010 Is = .33, p= .025
Effect of airline hubbing r= .35, p= .020 f= .27, p= .073
Weather r= .45, p= .002 f= .37, p= .012
Number of complex aircraft routings NS f= .29, p= .057
Number of restricted areas NS f= .24, p= .106
Requirements for sequencing and spacing NS f= .23, p= .129
Radio frequency congestion r=.49, p= .001 Tf, = .48, p= .001
Average traffic volume r=.33, p= .025 fe = .32, p= .033
Total 16CF score (sum) =.38, p= .010 fy = .36, p= .015
Average density NS r= .26, p= .090
Average complexity r= .42, p= .004 r= .51, p= .000

39



TABLE 4. SUMMARY OF LITERATURE REVIEW (CONTINUED)

Source

Factors or Issues

Grossberg (1989)

Sector complexity factors include control adjustments to merge
and space aircraft, climbing and descending aircraft flight paths,
mixture of aircraft types, frequent coordination, and heavy traffic.

Mogford, et al. (1993)

ATC complexity factors that may affect controller workload (and

OEs):

1. Number of climbing and descending aircraft.

2. Degree of aircraft mix.

3. Number of intersecting flight paths.

4. Number of multiple functions controller must perform.

5. Number of required procedures controller must perform.

6. Number of military flights.

7. Frequency of contacts (coordination) or interface with other

entities.
Extent to which controller is affected by airline hubbing.
Extent to which controller is affected by weather.

. Number of complex aircraft routings.
. Extent to which controller is affected by restricted, warning,

and military operating areas.

. Size of sector airspace.

. Requirement for longitudinal sequencing and spacing.
. Adequacy and reliability of radio and radar coverage.
. Amount of radio frequency congestion.

. Average amount of traffic.

Rodgers and Manning (1995)

OE time period shows increases in sector transit time, handoff
acceptance latency, vertical separation, and aircraft density and
decrease in number of handoffs accepted.

37





conditions may result in increased turbulence, icing,
and storm activity that effectively reduces the amount
of airspace available for flights and may result in
runway closings. Fowler (1980) thought that such
sector-related factors could contribute to OEs and
accidents.

The following two studies used simulation-based
experimentation to analyze the factors affecting con-
troller workload and performance. They provide an-
other viewpoint in the search for evidence of the
influence of ATC complexity on OEs.

Buckley, DeBaryshe, Hitchner, and Kohn (1983)
performed two experiments to assess the feasibility of
using dynamic real-time simulation procedures for
testing ATC systems. The purpose of the work was “to
determine the quality of measurement of system per-
formance and statistical treatment that is possible and
appropriate in dynamic simulation of air traffic con-
trol systems” (p. 1). The studies identified the impor-
tant basic dimensions for measuring ATC functions
in real-time dynamic simulations. Of interest to the
topic of OEs is that the authors addressed the issue of
the effect of sector geometry and traffic density on
various controller performance measures.

The first experiment examined the effects on per-
formance of two en route sector geometries and three
traffic levels ranging from very light to very heavy.
Data were collected from two, 1-hour runs for each of
31 controllers. The results of this experiment led the
researchers to conduct a less complex experiment
using only one of the possible six combinations of
conditions of sector and geometry. This second ex-
periment examined the effects of replication and pro-
vided a sufficient amount of data to enable the
completion of a factor analysis. Twelve, 1-hour runs
were conducted using the same sector with the same
traffic level for each of 39 controllers.

One of the outcomes of the first experiment was
that there was a statistically significant effect of sector
geometry and traffic density on almost all of the 10
performance measures. There was also a significant
interaction effect between geometry and density.
Buckley, et al. (1983) suggested that “Sector [geom-
etry] and [traffic] density are, as expected, important
factors in determining the results which will occur in

a given experiment, but they interact in a complex
way. The nature and extent of this interaction depend
upon the measures involved” (p. 73). This research
provides evidence that both traffic and sector factors
may interact to affect controller performance and,
presumably, the possibility of OE events.

Stein (1985) conducted a simulation experiment to
determine the relationship between a number of air-
space factors and controller workload. Workload was
measured by the Air Traffic Workload Input Tech-
nique (ATWIT) in which the controller pressed 1 of
10 buttons on a console with 1 representing low
workload and 10 representing high workload.

Ten air traffic controllers participated in a series of
one-hour simulations. Subjects experienced a low,
moderate, or high task load as defined by the number
of aircraft in a sector and the clustering of aircraft in
a small amount of sector airspace. Controller input to
ATWIT was performed once per minute. Stepwise
regressions were done using ATWIT scores as a crite-
rion measure. Four variables produced a multiple
correlation of R = .85 with the workload measure.
These were (in order of entrance into the stepwise
multiple regression equation) clustering of aircraft in
a small amount of sector airspace, number of hand-
offs outbound, total number of flights handled, and
number of hand-offs inbound.

The study demonstrated a strong relationship be-
tween controller workload and a subset of airspace-
and traffic-related variables. In addition, controllers
were able to provide real-time workload estimates
using the ATWIT without any noticeable decrement
in performance. Workload was best predicted through
a multivariate combination of airspace variables. The
factors listed were used to guide research into error
causation at the Atlanta ARTCC.

Grossberg (1989) and Mogford, et al. (1993) con-
ducted research to investigate the factors comprising
ATC complexity. Grossberg (1989) found a statisti-
cally significant relationship between sector complex-
ity, as defined by FAA Order 7210.46, and the rate of
OE incidence at the Chicago ARTCC. The correla-
tion was statistically reliable, but low in magnitude.
This provided an impetus for obtaining more infor-
mation on factors that affect sector complexity.


TOTALOE

TOTCOMX

VICTOR

VOL

VORTAC

(Coefficient / (Cases)

". " is printed if a coefficient cannot be computed

AVG_COM_

-4189
45)
-004

6951
45)
.000

.1124
45)
+462

-5943
45)
-000

+0307
45)
+842

~5641
45)
-000

AVG_DEN_

P=

Correlation Coefficients

+2108
45)
-165

2521
45)
.095

-4612
45)
-001

+2614
45)
- 083

+3663
45)
+013

+1766
45)
+246

CLDC

+3259
45)
+029

+7285
45)
+000

+5220
45)
-000

.3938
45)
007

-0800
45)
602

+4149
45)
.005

COOR

P=
P=
(

P=
(

P=
P=

(
P=

/ 2-tailed Significance)

A3

.1482
45)
.331

4835
45)
-0021

-0198
45)
898

-1560
45)
+306

+0011
45)
994

+3467
45)
+020

CUVOL

-.1354
( 45)
P= .375

-.2623
( 45)
P= .082

+7778
( 45)
P= .000

-.0806
( 45)
P= .599

+2872
(45)
P= .056

-.2065
( 45)
P= .174

FPAPCT

(
P=

-1073
45)
+483

-1533
45)
+315

.1142
45)
2455

+0023
45)

= .988

-0075
45)
961

-1607
45)

= .292


TOTALOE

TOTCOMX

VICTOR

VOL

VORTAC

MIX

-.2827
( 45)
P= .060

- 0646
( 45)
P= .673

~.6223
(45)
P= .000

-.1948
( 45)
P= .200

-1241
( 45)
P= .417

-.1692
( 45)
P= .266

NOINTSX

P=

P=

Correlation Coefficients

1273
45)
-405

.2160

45)
-154

- 4634

45)
+001

0138

45)
928

+3588

45)
.015

-1482

45)
-331

OTHRTS

P=

0004
45)
-998

+0137

45)
~929

-9054

45)
+000

+2226

45)
+142

+2923

45)
051

-1434

45)
.347

PROC

(

(
P=

(
P=

P=

P=

P=

(Coefficient / (Cases) / 2-tailed Significance)

+3818
45)
.010

8419
45)
.000

- 0583
45)
+704

-5653
45)
000

-0470
45)
+759

5197
45)
000

" , " is printed if a coefficient cannot be computed

AQ

RAD

0129
45)
~933

-1291

45)
-398

.1542

45)

= .312
~2156

45)
-155

+2068

45)
.173

0904

45)
-555


analyses of variance (MANOVAs) were calculated for
the measures in Table 2. Significant differences be-
tween the period preceding the error and the error
interval were found for the variables in Table 3.

The results suggest controllers were busier during
the error period than the preceding period. The data
also show that, although there was greater vertical
separation during the error period, aircraft density
may have increased, as indicated by increased time
within the criterion distance.

Given these results, it is not certain how ATC
complexity factors may have been operating in these
incidents. However, the findings shown in Table 3
(especially with regard to handoffs) could be consis-
tent with coordination problems, a theme echoed by
other authors in this literature review.

3.3 Discussion

Given the preceding review, it is helpful to summa-
rize factors from the literature that are relevant to the
relationship between OEs and sector or traffic fea-
tures, as found in Table 4.

The most relevant observation to make about the
above summary is how little research has focused
directly on the topic of interest: the relationship of
sector characteristics to OE occurrence. The only
author who directly addressed this was Grossberg
(1989). He found that some sector and traffic charac-
teristics (or ATC complexity factors) were correlated
with errors. These included control adjustments to
merge and space aircraft, climbing and descending
aircraft flight paths, mixture of aircraft types, frequent
coordination, and heavy traffic. Even in this author’s
report, it is difficult to determine exactly which fac-
tors were found to be correlated with OEs.

One reason for the absence of work on this topic
may be due to the tendency for error-reporting sys-
tems to classify errors ata high level without providing
for an analysis of causality. A very comprehensive
review of the FAA OE database by Kinney, et al.
(1977) found that most OEs were attributed to con-
troller attention, judgment, or communication prob-
lems. However, it was not feasible, given these error
categories, to determine if controller, CHI, or sector
design problems were involved. It is impossible to
conduct a deeper analysis of such data after the fact;

the OE reporting system used at the time of the error
defines the bounds of the information available. Other
efforts to classify OEs by Schroeder (1982), Stager
and Hameluck (1990), Stager, et al. (1989), Redding
(1992), Schroeder and Nye (1993), and Rodgers and
Nye (1993) have had to deal with similar issues in
reviewing OE database information.

Without additional information concerning the
percentage of time ATCSs spend controlling traffic
under various complexity or workload conditions, it
is difficult to determine the primary factors associated
with these outcomes. Unfortunately, without norma-
tive data, one must settle for a description of the
factors associated with operational irregularities. Ad-
ditionally, the reporting process, including the re-
porting reliability of the investigators, may affect the
extent to which these relationships can be determined.

In spite of these limitations, the data suggest some
possibilities about the relationship of ATC complex-
ity factors and OEs. Kinney etal. (1977) observed that
2% of the contributing factors listed in the 1974-76
SEIS reports were listed as problems with procedures.
Although this term can be used to describe a variety of
activities, it can be interpreted as referring to sector-
specific actions the controller is supposed to take with
regard to air traffic. However, only a low percentage of
contributing factors was associated with this category.

Another suggestion referencing or implying the
role of complexity factors is found in Schroeder (1982),
who implied that factors apart from traffic volume
must contribute to controller workload and, presum-
ably, to OE occurrence. Redding (1992) and Schroeder
and Nye (1993) found that coordination and misuse
of displayed data accounted for many errors.

Coordination is also a theme among other re-
searchers, such as Arad, et al. (1964), Couluris and
Schmidt (1973), Stein (1985), Fowler (1980),
Mogford, et al. (1993), and (indirectly) Rodgers and
Manning (1995). There may be a strong relationship
between OE occurrence and the amount of coordina-
tion required between sectors. The frequency of this
activity is largely determined by the location of sector
boundaries. If sector or facility boundaries are placed
near an intersection or area of heavy traffic flow,
controller workload and the probability of OEs could
be increased.


Technical Report Documentation Page

T. Report No. 2. Goverment Accession No. 3. Recipient's Catalog No.
DOT/EAA/AM-98/14

4, Title and Subtitle %. Report Date

The Relationship of Sector Characteristics to Operational Errors May 1998

6. Performing Organization Code

7. Author(s) &. Performing Organization Report No.
Rodgers, M.D.', Mogford, R.H.’, and Mogford, L.S.’

‘9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

"FAA Civil Aeromedical Institute, P.O. Box 25082, Oklahoma City, OK 73125
(Now at Office of Aviation Research, Federal Aviation Administration
Headquarters, AAR-100, Washington, DC 20591.)

*William J. Hughes Technical Center, Atlantic City, NJ 08405 17. Gontract or Grant No

*Rigel Associates, 9 Greenbriar Road, Marmora, NJ 08223 DTFA-02-95P35434

72, Sponsoring Agency name and Address 13. Type of Report and Period Covered
Office of Aviation Medicine

Federal Aviation Administration 14. Sponsoring Agency Code

800 Independence Ave., S.W.
Washington, DC 20591

15. Supplemental Notes

16. Abstract

An exploratory study was conducted on the relationship of air traffic control (ATC) complexity factors to operational
errors (OEs). This consisted of a detailed examination of OE data from 1992 through 1995 from the Atlanta en route
center. The Systematic Air Traffic Operations Research Initiative (SATORI) system was used to collect data for the
analysis. Sectors were categorized into zero-, low-, and high-error groups. Fifteen sector and traffic flow variables had
statistically significant correlations with OE frequency. Four variables were higher for the high-error group as compared
to the zero-error group. Sector size was smaller for the high-error group as compared to the combined zero- and low-
error categories. A significant multiple correlation was found between overall OE rate and a subset of the ATC
complexity measures. The data were also analyzed to define relationships between the complexity measures and
controller situational awareness (SA) at the time of the OE. The only statistically significant difference between OEs
with and without SA was for horizontal separation. In addition, high-error sectors were characterized by low SA for
errors. Certain sector and traffic flow characteristics were associated with these high-error sectors, suggesting that these
factors may negatively affect SA. It was concluded that the results demonstrated a relationship between sector
complexity and OE rate. Such findings, if extended, could assist with traffic management, sector design activities, and
the development of decision-support systems.

17. Key Words 18. Distribution Statement
Air Traffic Control, Sector Complexity, Controller Error, | Document is available to the public through the
Operational Errors National Technical Information Service,
Springfield, Virginia 22161
19. Security Classif. (of this report) 20. Security Classi. (of this page) 21. No, of Pages 22, Price
Unclassified Unclassified G7

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized



Number of OEs

35

1 2 3 4 5
Complexity

FIGURE 11. DISTRIBUTION OF OEs BY COMPLEXITY RATING

Number of OEs

357

4 3 2 -1 0 1 2 3 4
Workload Index (z score)

FIGURE 12. DISTRIBUTION OF OEs BY WORKLOAD INDEX

30



A factor analysis of 26 of the sector characteristic
variables indicated 3 possible underlying dimensions:
Traffic Activity, Size, and Military. Three further
factors were identified but did not account for much
of the original information. However, the annual
review measure of average traffic density did not
appear to be closely related to other estimates of traffic
volume (such as 16CF ratings of frequency congestion
and traffic volume), suggesting that it may not be
performing as expected.

A review of the OE database for the 85 errors
collected between 1992 and 1995 shows that 81% of
the overall causal factors were attributed to problems
with the radar display, 29% were assigned to commu-
nication errors, and 11% to coordination.

OEs at the Atlanta ARTCC during the reporting
period mostly took place between 0800 and 2000 hr
and concentrations were found between 15,000 and
20,000 ft and between FL300 and FL350. Given that
separation standards change at FL290, the flight levels
of OEs tended to be distributed into two groups, accord-
ing to the zone in which they occurred. A disproportion-
ately high number of errors occurred in high sectors, with
fewer than expected in ultra high sectors.

Mean traffic volume at the time of an OE was
significantly higher than as reported by the 1995
annual review of average sector density. The average
density of 8.0 aircraft when an error occurred was
nearly one standard deviation above the base rate level
of 6.5 aircraft. Judgments of complexity at the time of
an OE averaged 3.4 (on a scale of 5), and most errors
were rated as occurring in moderately complex condi-
tions. A workload measure derived from the traffic
density and complexity data indicated that some er-
rors occurred at above average workload levels, as
compared to the whole set of OEs. Normative or
baseline workload data for Atlanta ARTCC sectors
were not available for comparison.

The minimum horizontal and vertical separation
distance at the time of the error were further analyzed
by considering RMS distances. Although there are
different vertical separation requirements above and
below FL290, the 1000 ft of additional separation
only creates another 50 ft of straight line distance.
Most errors were found to have at least 20,000 ft (or
about 3.3 mi) of RMS distance remaining.

Conflict severity was calculated for the OEs and it
was found that 84% were moderate and 16% were
severe. There was a moderate correlation between
sector type (or level) and severity, suggesting that
more severe errors occurred in sectors below FL230.

Most OEs happened with one or two controllers
working. Without normative information, it is diffi-
cult to know the proportion of time that one, two, or
three controllers typically work a position. In 73% of
the cases, the radar controller was not aware of the
developing OE.

Sectors in the Atlanta ARTCC were divided in no-
error, low-error, and high-error groups. Bivariate cor-
relations were calculated between most of the 29
original sector characteristic variables and the OE
frequency measures (number of OEs and OE group).
Fifteen variables had correlations at p < .1 with OE
group. Given a significant MANOVA, further analy-
ses determined that there were statistically significant
differences between the no-error and high-error groups
on four sector variables including frequency of prob-
lematic weather, radio frequency congestion, total
complexity, and average complexity. There were mar-
ginally significant differences (indicating possible
trends in the data) for amount of climbing/descend-
ing traffic, degree of aircraft mix, number of required
procedures, and average traffic volume. In general,
these results suggested that sectors with high error
counts were more complex than those with no errors.

All measures in the ANOVA increased as a function
of error count, except for aircraft mix which de-
creased. A more homogenous (low mix) traffic pattern
composed of high-speed jets could pose more chal-
lenges for maintaining separation.

While sector size (as measured by OpenCreate) did
not emerge as a significant variable in the ANOVAs,
inspection of the size differences between OE fre-
quency groups showed a relationship, and this was
supported by a statistical test. It is clear that high-error
sectors were only about 32% of the size of no- and
low-error sectors. More errors occurred in lower,
smaller sectors. It is not surprising that smaller sector
size would induce more complexity.

ANOVAs were used to consider the differences in
OE characteristics between low and high OE sectors.
First, correlations were found between OE group (low


NOTICE

This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents or use thereof.


TABLE 13. RELATIONSHIP OF FINDINGS TO LITERATURE REVIEW (CONTINUED)

Source Factors or Issues Relevant Factors
Fowler (1980) Sector complexity effected by problems with Correlations with OE rate:
coordination, procedures, LOAs, and weather. Complex routings
Weather problems higher in high
OE sectors.
Buckley, et al. Sector geometry and traffic density interact to Correlations with OE rate:
(1983) affect controller performance. Traffic volume,
Complex routings,
Climbing/descending traffic.
Stein (1985) Controller workload related to clustering of aircraft | Correlations with OE rate:
in a small amount of airspace, number of hand- Traffic volume.
offs outbound/inbound, and total number of flights | High error sectors were smaller.
handled.
Grossberg (1989) | Sector complexity factors include control Correlations with OE rate:

adjustments to merge and space aircraft, climbing
and descending aircraft flight paths, mixture of
aircraft types, frequent coordination, and heavy
traffic.

(Sequencing/Spacing NS),
Climbing/descending traffic,
(Traffic mix NS),

Traffic volume.

Mogford, et al.
(1993)

ATC complexity factors that may affect controller

workload (and OEs):

Number of climbing and descending aircraft.

Degree of aircraft mix.

Number of intersecting flight paths.

Number of multiple functions controller must

perform.

Number of required procedures controller

must perform.

Number of military flights.

Frequency of contacts (coordination) or

interface with other entities.

8. Extent to which controller is affected by airline
hubbing.

9. Extent to which controller is affected by
weather.

10. Number of complex aircraft routings.

9 FEN

No

Correlations with OE rate:

1. Climbing/descending traffic,
(Traffic mix NS),

3. (Intersections NS),

4. Multiple functions,

5. Procedures,

6. (Military traffic NS),

7. (Coordination NS),

8. Hubbing,

9. Weather,

10. Complex routings,

11. Restricted areas,

12. (Size NS),

13. (Sequencing/spacing NS),

14. (Radio/radar coverage NS),

15. Frequency congestion,

16. Traffic volume.

Mogford, et al. 11. Extent to which controller is affected by High OE sectors have more:
(1993) restricted areas, warning areas, and military Frequency congestion,
operating areas. Problem weather,
12. Size of sector airspace. Total complexity.
13. Requirement for longitudinal sequencing and | Sectors smaller in high OE sectors.
spacing.
14. Adequacy and reliability of radio and radar
coverage.
15. Amount of radio frequency congestion.
16. Average amount of traffic.
Rodgers and OE time period shows increases in sector transit Aircraft volume higher at time of
Manning (1995) time, handoff acceptance latency, vertical OE.

separation, and aircraft density and decrease in
number of handoffs accepted.

42



Number of Sectors

5000 10000 15000 20000 25000 30000 or more

Cubic Volume (cu mi)

FIGURE 3. DISTRIBUTION OF SECTORS BY CUBIC VOLUME OF AIRSPACE

Number of Sectors

10

High Ultra High
Sector Type

FIGURE 4. DISTRIBUTION OF SECTORS BY TYPE

26


TABLES

TABLE 1. NUMBER OF OPERATIONAL ERRORS AS A FUNCTION OF TRAFFIC
VOLUME AND WORKLOAD COMPLEXITY IN 1974-76 SEIS DATA

Workload Complexity
Traffic Volume Light Moderate Heavy Total
Light 127 68 9 204
Moderate 22 160 70 252
Heavy 0 14 73 87
Total 149 242 152 543

TABLE 2. MEASURES USED IN THE RODGERS AND MANNING (1995) STUDY

Measure Type Variable
Sector Activity Handoff Activity Number of Aircraft Transiting
Sector
Average Sector Transit Time
Number of Handoffs
Accepted
Latency to Accept Handoffs
Host computer system Number of HCS Inputs
(HCS) Inputs Number of Input Errors
Aircraft Proximity Conflict Alert Number of Conflict Alerts
Duration of Conflict Alerts
Proximity Average Horizontal
Separation
Average Vertical Separation
Number of Aircraft Pairs
Within a Criterion Distance of
Each Other
Average Time Aircraft Pairs
Spent Within a Criterion
Distance of Each Other
Load Number of Aircraft in Sector
by Track Type
Aircraft Dynamics Speed, Altitude, and Average Change
Heading Number of State Changes
Number of Aircraft Changing
State
Number of Change Over a
Criterion Level

TABLE 3. MANOVA RESULTS

Variable Result
Average Sector Transit Time Increase
Average Latency to Accept Handoff Increase
Number of Handoffs Accepted Decrease
Average Vertical Separation Increase
Average Time Aircraft Were Within 10 mi & 1000 ft__| Increase

35


This result is nearly identical to a finding by Rodgers
and Nye (1993) that more severe errors occurred
below FL290.

A factor analysis using OpenCreate and sixteen
complexity factor (16CF) variables indicated three
primary underlying dimensions: Traffic Activity, Size,
and Military. The other three factors in the analysis
showed no strong patterns, with one factor lacking
any distinct loadings. It is reassuring to observe that
the average complexity facility review measure was
associated with other Traffic Activity variables and
that the sector size variables, as measured by
OpenCreate and 16CF, were related. However, the
facility review traffic density variable was not loaded
on the same factor as other traffic density indicators.
This suggests that it may lack concurrent validity.
Fortunately, all other volume-related measures were
loaded on the Traffic Activity factor. It would be
helpful to evaluate traffic density in the Atlanta
ARTCC sectors in additional ways to better define the
nature of the average density measure.

Traffic mix was associated with the Size factor,
rather than with Traffic Activity. The presence of a
negative weighting of aircraft mix on this factor sug-
gests that a preponderance of larger jet aircraft may be
associated with larger sectors. The collection of data
on aircraft type for each OE would be useful in this
analysis. Adequacy of radio/radar coverage fell in with
the Military factor. Areas that have poor radio and
radar coverage have some similarity to military air-
space in that they are less accessible to FAA air traffic
control. Finally, the sector shape measure, percent of
cube, seemed to have no relationship to other measures.
This could indicate that it taps some independent factor.
However, by accounting for only 7% of the variance in
the data set, it may be of limited importance.

The potential for a reduction in sector measures to
a set of three or four factors holds promise in that it
might simplify the process of evaluating sector char-
acteristics. The practice would be to use one or more
of the variables that were loaded on each distinct
factor. However, given that these variables do not

fully represent each factor, some information is lost.
Further research would be needed to determine the
utility of this approach.

A theoretical issue worth noting is that, given the
wide range of ATC complexity measures available,
there appear to be only three general characteristics of
sectors related to traffic activity, size, and military
operations. This makes intuitive and practical sense,
given controllers’ descriptions of sector workload
issues. The level of difficulty of an air traffic situation
is often described as an interaction between sector size
and the amount and behavior of traffic. This relation-
ship was also found in the Buckley et al. (1983) work
where sector geometry and traffic density interacted
to affect controller performance.

Stein’s (1985) research showed that variables re-
lated to the traffic activity and size factors accounted
for a large proportion of controller workload. In the
current study, Military was discovered to be an addi-
tional independent factor. This seems reasonable in
that military aircraft and their associated airspace
reservations are controlled by outside agents and must
be accommodated by the controller in different ways
than are commercial aircraft, weather, and other centers.

An application of the factor analysis findings could
be used as a general guideline for traffic management
systems. Aircraft activity, sector size, and military
operations should be included in any formula that
seeks to account for or predict ATC complexity.

When comparing sectors with no errors, less than
three errors, and four or more errors, there was consis-
tent evidence supporting the role of ATC complexity.
First, 15 of the sector and traffic-related variables were
correlated with OE frequency or OE frequency group."
Also, it was found that there were differences between
OE frequency groups on a number of measures. Five
variables (including sector size) showed statistically
significant differences between groups, and two of
these were general complexity measures. Three other
variables demonstrated trends in the same direction
(increasing errors with higher complexity). This demon-
strates a definite role for ATC complexity in OE analysis.

° Although many of the correlations found were in the .30 range, and therefore not particularly strong, many were statistically significant.
Correlations in this range are acceptable for this type of exploratory research.


(6%), operations management (5%), environment
(such as receipt of erroneous information or lack of
compliance from pilots, user equipment failures, or
heavy controller workload) (1%), procedures (2%),
external (9%), and no code (6%).

Only one of the error categories (procedures) re-
lates to sector factors. This was subdivided into six
sub-categories according to whether the procedure
was inadequate, too complex, impractical, etc.

Kinney, et al. (1977) also examined system errors
in the context of reported controller workload. Traffic
volume and workload complexity at the time of occur-
rence of the error were subjectively rated as light,
moderate, or heavy. As Table 1 shows, most incidents
occurred under conditions of light or moderate vol-
ume and moderate workload complexity in the 1974-
76 SEIS data.

Other findings by Kinney, et al. (1977) showed no
relationship between errors and previous errors by the
same controller or between errors and controller age.
There was an indication in the 1976 data that control-
lers with more than 5 but less than ten years of
employment with the FAA experienced more prob-
lems. Controllers with less than 24 months of work
history in their current positions were slightly more
error prone. Finally, of those errors reported in en
route centers, 54% involved aircraft in level flight,
26% were climbing, and 20% were descending. These
results tend to support (by a few percentage points)
Siddigee’s (1973) assumption that most en route
conflicts should be in the horizontal dimension.

Schroeder (1982) examined the causes of loss of
separation and reviewed the FAA’s OE database (SEIS)
for the years 1970 through 1980. Errors per million
operations in terminal airspace steadily increased dur-
ing this period. However, the average error rate re-
mained stable for en route airspace. A detailed analysis
of the SEIS data for 1977 and 1978 was conducted. As
found by Kinney, et al. (1977), most errors occurred
under light or moderate workloads. Schroeder (1982)
identified an apparent shift in error patterns from
1965 and 1966 when most errors occurred under
moderate or heavy workload conditions.

Schroeder (1982) noted, “Thus, while traffic vol-
ume and the complexity of the airspace system have
increased significantly, a higher percentage of errors

involve light to moderate workloads” (p. 261). He
also observed that workload levels rated by the con-
trollers did not necessarily depend entirely on traffic
density. “There are obviously other aspects of the
situation that become involved in the determination
of this workload measure other than traffic volume
alone” (p. 261). Perhaps these other aspects could be
traffic flow or sector characteristic variables.

Kinney, et al. (1977) had found that most errors
were attributed to attention, judgment, and commu-
nications. However, the recording system changed in
1978, and different error categories were adopted.
Most OEs in the 1978 and 1979 data were blamed on
a failure of the controller to initiate corrective action.
Some errors were caused by omitting to coordinate
with other controllers as aircraft crossed sector bound-
aries. Schroeder (1982) noted: “In fact, a review of
error records from 1969 through 1980 indicates that
coordination was either a direct or contributing factor
in 27.3% to 53.9% of the errors for both en route and
terminal airspace” (p. 264). It may be that the control-
lers were not clear about the current sector configura-
tion in their areas or that sector design created an
unacceptable coordination demand. Other important
factors were flight data and communication prob-
lems. Schroeder’s (1982) research suggests that as-
pects of sector design may affect the way aircraft
transition between sectors and create the conditions
for OE occurrence.

Two studies on OEs in the Canadian ATC system
were conducted by Stager and Hameluck (1990) and
Stager, Hameluck, and Jubis (1989). They found that
operating irregularities occurred under low to moder-
ate workload conditions with none to normal com-
plexity. Primary error categories included attention,
judgment, and communication. Like Kinney, et al.
(1977), Stager, et al. (1989) distinguished between
direct and contributing causes with regard to control-
ler errors. Direct causes “refer to direct actions or the
failure to act on the part of the controller that results
inescapably in a loss of separation given a certain
system state” (p. 44). Contributing causes “refer to the
specific states of the controller (i.e., fatigue, distrac-
tion, attitudes, excessive workload, and procedural
knowledge) as well as the states of the environment,


- - Correlation Coefficients - -

RING SHELF SIZE SQ_SP TOTALOE TOTCOMX

MIL +1385 «2802 -0560 -.1842 0418 +2320
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .364 P= .062 P= .715 P= .226 P= .785 P= .125

MIX -.1618 +5947 -.3966 -.1417 -.2827 -0646
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .288 P= .000 P= .007 P= .353 P= .060 P= .673

NOINTSX -1139 +5237 -.3113 -.0713 -.1273 -2160
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .456 P= .000 P= .037 P= .641 P= .405 P= .154

OTHRTS +2106 -.3032 -6077 -0907 -.0004 -.0137
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .165 P= .043 P= .000 P= .554 P= .998 P= .929

PROC -5460 -0297 -.1728 +5229 -3818 ~8419
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .000 P= .846 P= .256 P= .000 P= .010 P= .000

RAD -2088 -.4060 0071 +3489 -.0129 +1291
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .169 P= .006 P= .963 P= .019 P= .933 P= .398

REST -.1153 .2873 -.1132 -.2600 .1946 +1390
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

= 451 P= .056 P= .459 P= .085 P= .200 P= .362

RTNG 1.0000 -.0880 -.1130 -5172 +2363 -6004
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P=. P= .566 P= .460 P= .000 P= .118 P= .000

SHELF -.0880 1.0000 -.1619 -.3071 -.0684 - 0283
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .566 P=. P= .288 P= .040 P= .655 P= .854

SIZE -.1130 -.1619 1.0000 -0020 -.1221 -0108
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .460 P= .288 P= . P= .990 P= .424 P= .944

SQ_SP -5172 ~.3071 -0020 1.0000 .0929 -6180
( 45) ( 45) ( 45) ( 45) ( 45) ( 45)

P= .000 P= .040 P= .990 P=. P= .544 P= .000

(Coefficient / (Cases) / 2-tailed Significance)

" , " is printed if a coefficient cannot be computed

All


TABLE 14. COMPARISON OF OE CAUSES

Causal Factor

1985 to 1988 Data

1992 to 1995 Data

Radar Display

57%

81%

Communication

30%

29%

Coordination

30%

11%

Data Posting

20%

6%

Relief Briefing

4%

NA

43



Redding, R.E. (1992). Analysis of operational errors
and workload in air traffic control. Proceedings of
the Human Factors Society 36th Annual Meeting
(pp. 1321-5), Vol. 2. Santa Monica, CA: Human
Factors Society.

Rodgers, M.D. & Duke, D.A. (1993). SATORI: Situa-
tion Assessment Through Re-creation of Incidents.
(Report No. DOT/FAA/AM-93/12). NTIS No.
ADA268390. Washington, D.C: Federal Avia-

tion Administration.

Rodgers, M.D., & Manning, C.A. (1995, April). Baseline
assessment of the national airspace system: An ap-
proach toward modeling operational errors. Pro-
ceedings of the 8th International Symposium on
Aviation Psychology (pp. 422-5). Columbus, OH:
The Ohio State University.

Rodgers, M.D., & Nye, L.G. (1993). Factors associated
with the severity of operational errors at air route
traffic control centers. In M.D. Rodgers (Ed.), An
Examination of the Operational Error Database for
Air Route Traffic Control Centers. (Report No.
DOT/FAA/AM-93/22). pp. 11-25. NTIS No.
ADA 275986. Washington, DC: Federal Aviation
Administration.

Schmidt, D.K. (1976). On modeling ATC workload
and sector capacity. Journal of Aircraft, 13(7), 531-7.

Schroeder, D.J. (1982). The loss of prescribed separa-
tion between aircraft: How does it occur? Proceed-
ings (P-114), Behavioral Objectives in Aviation
Automated Systems Symposium (pp. 257-69). Wash-
ington, DC: Society of Automotive Engineers.

46

Schroeder, D.J., & Nye, L.G. (1993). An examination
of the workload conditions associated with opera-
tional errors/deviations at Air Route Traffic Con-
trol Centers. In M.D. Rodgers (Ed.), An examina-
tion of the operational error database for Air Route
Traffic Control Centers, (Report No. DOT/FAA/
AM-93/22). pp. 1-9. NTIS No. ADA275986.
Washington, DC: Federal Aviation Administration.

Siddigee, W. (1973). A mathematical model for pre-
dicting the number of potential conflict situations
at intersecting air routes. Transportation Science,

7(2), 158-67.

Siddigee, W. (Winter, 1973-74). Air route capacity
models. Navigation, 20(4), 296-300.

Stager, P., & Hameluck, D. (1990). Ergonomics in air
traffic control. Ergonomics, 33, 493-9.

Stager, P., Hameluck, D., & Jubis, R. (1989). Underly-
ing factors in air traffic control incidents. Proceed-
ings of the Human Factors Society 33rd Annual
Meeting (pp. 43-6), Vol. 2. Santa Monica, CA:
Human Factors Society.

Stein, E. (1985). Air traffic controller workload: An ex-
amination of workload probe (Report No. DOT/
FAA/CT-TN84/24). Atlantic City, NJ: Federal

Aviation Administration.

US Congress, House Committee on Government Op-
erations. (1976). Human errors and related defi-
ciencies in FAA air traffic control system: 27th report
(Report No. TL 740.8 1976u). Washington, DC:
US Government Printing Office.


Ninety-seven controllers rated the degree to which
12 factors contributed to the difficulty or complexity
of operations in their particular sector or area of
specialization. The complexity factors most frequently
cited in the Chicago ARTCC included: control ad-
justments involved in merging and spacing aircraft,
climbing and descending aircraft flight paths, mixture of
aircraft types, frequent coordination, and heavy traffic.
Sector-related factors, such as large sector airspace and
intersecting flight paths, received lower ratings.

Grossberg combined the factors with the four high-
est ratings to form a complexity index. He found that
this index was correlated with the number of OEs
found in sectors in the Chicago ARTCC. Data were
collected for 21 months in 1987 and 1988. The
complexity index was highly correlated (r = .74) with
frequency of OEs. Correlations between the standard
FAA formula and the same OE database correlations
were not as high (r = .44).

Mogford, et al. (1993) conducted a study to exam-
ine the cognitive processes associated with ATC.
Controllers from the five specialization areas in the
Jacksonville ARTCC participated. The purpose of the
research was to identify complexity factors and compare
the use of direct (questionnaire and interview) versus
indirect (statistical) methods for factor identification.

Direct methods included asking controllers to sug-
gest and then rate complexity factors in terms of how
they made sectors more or less difficult to control.
Indirect methods involved having controllers make
paired comparisons with respect to complexity be-
tween maps of sectors in five specialization areas.
Multidimensional scaling (MDS) was used to formu-
late complexity factors by determining whether the
arrangement of sectors along each MDS axis corre-
sponded to the increase or decrease in some variable or
factor related to complexity.

Thirteen of the 19 total complexity factors were
produced by both methods, showing a close corre-
spondence between direct and indirect techniques for
determining ATC complexity factors. The 19 vari-
ables were regressed over an overall complexity crite-
rion formed by ratings of five Traffic Management
Unit staff members who were familiar with all sectors
in the ARTCC. The factors of complex aircraft

routings, spacing and sequencing for departures and
arrivals, and frequency congestion during peak peri-
ods formed a significant multiple correlation (R = .85)
with the overall complexity criterion.

After further analysis, the factor definitions were
refined and some redundancies removed. The follow-

ing 16 unique ATC complexity factors were identi-
fied:

Number of climbing and descending aircraft.
Degree of aircraft mix.

Number of intersecting flight paths.

Number of multiple functions controller must per-

RYN

form.

5. Number of required procedures controller must
perform.

6. Number of military flights,

7. Frequency of contacts (coordination) or interface
with other entities.

8, Extent to which controller is affected by airline
hubbing.

9. Extent to which controller is affected by weather.

10. Number of complex aircraft routings.

11. Extent to which controller is affected by restricted
areas, warning areas, and military operating areas.

12. Size of sector airspace.

13. Requirement for longitudinal sequencing and spacing.

14. Adequacy and reliability of radio and radar coverage.

15. Amount of radio frequency congestion.

16. Average amount of traffic.

Although not specifically addressing sector design
issues, recent work by Rodgers and Manning (1995)
incorporating SATORI measures into a study of OE
occurrence at Atlanta ARTCC is relevant for this
review. Data for 12 OEs that occurred between 1992
and 1994 were analyzed using the Performance and
Objective Workload Evaluation Research (POWER)
system, a subroutine of SATORI. (These are a subset
of the data that were the focus of the current research
project.) POWER supports the collection of a variety
of air traffic and sector measures.

Seventeen minutes of data were collected on
each incident, 8% minutes preceding the error and
8% minutes during the error interval. Multivariate