Federal Aviation
Administration

Aviation Infrastructure Risk
Assessment: Effect ofCommunication and Surveillance
Facility Service Outages on
Traffic Separations
EIWAC 2013
February 20, 2013
Sherry S. Borener, Ph.D. (FAA)
Vitaly S. Guzhva, Ph.D. (MCR Federal)

Federal Aviation
Administration2

Background
Measuring safety as an outcome variable in a
High Reliability Organization is difficult anddoes not adequately capture the true safety
state of the system

Proactive approach is based on early
identification, assessment, and mitigation of any
credible hazards

Historic (forensic) approach cannot reveal
emerging future hazards

System Safety
ManagementTransformation(SSMT)
System Safety Assessment (SSA) uses risk
modeling and forecasting capability to identify
potential risk issues
SSA examines historical causes of events andpotential future exposures to develop
appropriate risk models

This study investigates how unscheduled
service outages affect traffic separations
Unscheduled service outages are precursors
that may lead to unsafe outcomes

Federal Aviation
Administration

3

Integrated Safety Assessment
Model (ISAM)

Federal Aviation
Administration

4

Midair Collision (MAC):
ASAP Safety Concept Mind Map - Similarity to ESD•Accident
Category•SafetyEnhancement•ProblemStatement•Safety Concept
MetricFederal Aviation
Administration5

Aircraft on Collision Course: ESD US-31
ESD:
31
Initiating event: Aircraft are positioned on collision course in flight
Flight phase: In flight

Colour coding
Historical NAS data
CATS data
Expert judgement
Calculation

Click Here to Return to Index

US31a1
Aircraft are
positioned oncollision course inflight
6.84E-05

US31b1ATC does not resolve
the conflict

3.09E-04
US31c1Flight crew does notresolve the conflict

2.77E-01

3.09E-04

US31a1

US31b1•Fault Tree X•Fault Tree Y•Ignoring TCAS
would have negative
impact on this Fault
Tree
END EVENTEVENT CODEFREQUENCY PERFLIGHT

Collision in mid-air

US31d1_01

5.85E-09

Aircraft continues flight

US31d2_02

1.89E-05

Aircraft continues flight

US31c2_03

4.95E-05

Total

6.84E-05

FREQUENCY PERFLIGHT (input)

5.85E-09

US31c1•Fault Tree Z

YesNo

Federal Aviation
Administration

6

Background
57,558 ATM
facilities
werefunctioning
in the NAS
in 201140,000 ofthem
experienced
unscheduled
serviceoutages

Impliedaverageannual
outage ratioof 0.69

What are the ATM safety implications?
Federal Aviation
Administration

7

Number of Outages vs. Number ofFacilities (2007-2011)
50000
40000
30000
20000
10000
02007

2008

2009

2010

10000
8000
6000
4000
2000
02011

2007

6000
5000
4000
3000
2000
1000
02007

2008

2009

2010

•Total unscheduled outages

2011

Namber of Facilities

Namber of Facilities

2.00
1.80
1.60
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00

7000

2008

2009

2010

2011

•Communication Facilities

Outage Ratio•Surveillance Facilities
8000

2.00
1.80
1.60
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00

2.00
1.80
1.60
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00

40000
35000
30000
25000
20000
15000
10000
5000
02007

•Total number of facilities

20082009

2010

Outage Ratio

60000

12000
Namber of Facilities

2.00
1.80
1.60
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00

Outage Ratio

Number of Facilities

70000•Navigation Facilities

Outage Ratio•All ATM Facilities

2011•Outage Ratios

Federal Aviation
Administration

8

Annual Outage Ratios
2007-2011 Average
2.00
1.80

1.75

1.60
1.44
1.40
1.20
1.00

0.92

0.80
0.63
0.60

0.51
0.36

0.40

0.12

0.20

0.09

0.00
Surveillance

Navigation

Weather

Environmental

Automation

Communication

Other

Telecommunication

Federal Aviation
Administration

9

Study Objectives

Overall• Examine safety impact of communication andsurveillance facilities service outages

Specific• Investigate impact of outages on TCAS RA andseparation events in the facility service volumes• Investigate impact of outages on the separationindex• Estimate marginal effects of outages for different
types of facilities

Federal Aviation
Administration

10

Sample Construction
Unscheduled Service Outage Data in the vicinity of 15 majortraffic hubs (2010–2011)• Source: National Airspace System Performance Analysis
System (NASPAS)• 222 Communication and 116 Surveillance Facility Outages

Traffic Separation Data for Facility Service Volumes• Radar track data for +/- 30 min of an outage. Source: Offload
Extract of Sector Design and Analysis Tool (SDAT)• Traffic separations estimated by ISA

TCAS RA Modeling by ISA

Federal Aviation
Administration

11

TCAS RA ModelingFederal Aviation
Administration

12

Separation Conformance
Categorization

Federal Aviation
Administration

13

ISA Separation ModelingFederal Aviation
Administration

14

ISA Separation ModelingFederal Aviation
Administration15ISA Separation ModelingFederal Aviation
Administration

16

Average TCAS RA Count
12TCAS RAs Per 1,000 Flights

1086420-30-25-20-15-10-505
Minutes From an Outage

101520Federal Aviation
Administration

253017

Actual TCAS RA Encounters
MIT Lincoln Labs 2009 Study
AirportRate per operation

Rate per 1,000 flights

EWR

1 in 90

11.1

JFK

1 in 291

3.4

LGA

1 in 93

10.8

ISP

1 in 107

9.3

CDW

1 in 934

1.1

FRG

1 in 125

8.0

HPN

1 in 34

29.4

MMU

1 in 90

11.1

TEB

1 in 28

35.7

Average

13.3

Federal AviationAdministration

18

Average CAT A Event Count30

Cat A Events Per 1,000 Flights

2520151050-30-25-20-15-10-505Minutes From an Outage

101520Federal Aviation
Administration

2530

19

Before and After
Average Counts
45
40
35
30
25
20
15
10
5042.438.8
31.6 31.9
19.5

22.9

"- 30 min""+ 30 min"

7.6 9.2

TCAS
35%
30%
25%
20%
15%
10%
5%
0%
-5%-10%

23.9

25.7

CAT A

CAT B

CAT C

CAT PE

31.5%
21.5%
9.2%% Change
1.0%TCAS

CAT A

CAT B

CAT C
-4.2%

CAT PE

Federal Aviation
Administration20

Analysis Methodology
Count data analysis• RA encounters; Loss of separation events• Negative Binomial regression modelContinuous data analysis• Separation index values• Lognormal regression model

Federal Aviation
Administration

21

Count Data Analysis
TCAS RA and Lost Separation Events Data
Count data that cannot be negative

TCAS RA and Lost Separation Events Data
Presence of over-dispersion

When variance greater than the mean

Negative Binomial Regression Model
Widely used in biomedical andhighway safety research

Should be controlled for traffic density
in the facility service volumes

Federal Aviation
Administration

22

Negative Binomial Regression
Model1⎞⎛1/αyiΓ⎜ yi + ⎟⎛⎞⎛⎞
1αµα⎝⎠ ⎜i⎟⎜⎟ •, yi = 0, 1, 2, 3, 4,p(Yi = yi ) = p( yi ) =Γ( yi + 1)Γ( α1 ) ⎜⎝ 1 + αµi ⎟⎠ ⎜⎝ 1 + αµi ⎟⎠•where µi = E (Yi ) = wi [exi′β∑ j=1 xij′ β jk] = wi [e] •, i =1,2,3,….,n.

Var (Yi ) = µi + αµ i2 •where α ≥ 0 – dispersion parameter•Specifically:SEi = a + b1OUTi + b2 SUROUTi + b3COUNTSi + b4VOLDISi + ei•

Where:––––––

SE is a separation event (TCAS RA, Cat A, B, C, or PE)
OUT is a dummy variable that indicates if the service was out or not
SUROUT is a dummy variable that indicates that the facility that lost service was a surveillance facility
COUNTS is the number of traffic counts in the facility service volume
VOLDIS is the total distance in nm that all flights flown in the facility service volume during examined period
b1 and b2 are coefficients of interest; b3 and b4 are coefficients of variables that control for traffic density

Federal Aviation
Administration

23

Negative Binomial Regression
Model Parameter Estimates
SEConst.

OUTSE OUTSE NOUTSUROUT

COUNTS

VOLDISTCAS RA-0.30*

0.27**

1.31

0.00

0.000056**

0.000017**

CAT A

-0.06

0.27**

1.31

0.00-0.000031

0.000064**

CAT B

2.14**

0.14**

1.15

0.00-0.000014

0.000018**

CAT C

1.60**

0.01

N/A

0.00

0.000018**

0.000009**

PE

1.76**

0.06**

1.06

0.00

0.000031**

0.000009**•* Indicates marginal (10%) statistical significance; ** Indicates statistical significance (5% or better)•Interpretation: coefficient of OUT indicates the difference of event count Logs. For example, for
TCAS RAs: Log(RA with Outage) – Log(RA without Outage) = 0.27. So, Log(RA Out/RA No Out)= 0.27, making (RA Out)/(RA no Out) = 1.31. TCAS RA encounters are 1.31 times more likely inthe service volume of the facility with service outage.

Federal Aviation
Administration

24

Results of the Separation EventsAnalysis
TCAS RA Encounters and Cat A separation events
are 1.31 times more likely in 30 minutes following aservice outage than in 30 minutes before an outage

Cat B events are 1.15 and PE events are 1.06
times more likely in 30 minutes following an outage.
Cat C events are not affected by service outages

Surveillance facility outages are not marginallydifferent from communication facility outages

Federal Aviation
Administration

25

Continuous Data Analysis
Separation Index Data
Ratio of actual minimum distance between aircraft at the same altitude to required
separation distance. For example, if the minimum distance between aircraft was 6 nm
when required separation was 5 nm, the separation index is 120%

Separation Index Data
Cannot be negative. Only data for aircraft within 10 nm of another aircraft is usedLognormal Regression Model
Typically used when dependent variablecannot be negative

Should be controlled for traffic density inthe facility service volumes
Federal Aviation
Administration

26

Lognormal Regression Model22′•whereyispositive′•and[][]
VarYσβx=′
E [Y ] = β x
Y = β x+e

E [LogY ] = Log (β ′x ) − 2σ 2 •and Var [LogY ] = σ 2•In thismodel•Specifically:

SI i = a + b1OUTi + b2COMOUTi + b3COUNTSi + b4VOLDISi + ei•

Where:––––––

SI is a separation index for aircraft within 10 nm of each other
OUT is a dummy variable that indicates if the service was out or not
COMOUT is a dummy variable that indicates that the facility that lost service was a communication facility
COUNTS is the number of traffic counts in the facility service volume
VOLDIS is the total distance in nm that all flights flown in the facility service volume during examined period
b1 and b2 are coefficients of interest; b3 and b4 are coefficients of variables that control for traffic density

Federal Aviation
Administration

27

Lognormal Regression Model
Parameter Estimates

SIConst.

OUTCOMOUT

COUNTS

VOLDIS

1.71**-0.19**

0.15**

0.00

0.000005**•** Indicates statistical significance (5% or better)•Interpretation: coefficient of OUT indicates how the separation index was affected when asurveillance facility lost service – the separation index decreases by 19%. The coefficient ofCOMOUT indicates how the separation index was affected when the facility that lost service wasa communication facility – the separation index increased by 15% comparing with a surveillance
facility outage. Adding 15% to -19% results in -4%. The separation index decreases by 4% whena communication facility loses service.

Federal Aviation
Administration

28

Results of Separation Index
Analysis
The separation index for flights within 10 nm of
each other is 19 percent lower in 30 minutes
following a surveillance facility outage than in
30 minutes before an outage

The separation index is only 4 percent lower
for communication facility outages

Federal Aviation
Administration29

Next Steps
Surface surveillance
equipment outages(ASDE, ASDE-X) andRW incursions• More than 12,000 runway incursions in the US from 2001to 2013
• Do ASDE (-X) outages contribute to the likelihood ofrunway incursions?

ISAM(Integrated SafetyAssessment Model)• ISAM depicts a complete risk picture and incorporates allof the aspects related to safety hazards collectively,including equipment outages• Event Sequence Diagrams (ESDs) are quantified using
US data• Fault Trees (FTs) are used to model initial and pivotal
events in ESDs
• Service outages will be integrated in FTs

Federal AviationAdministration30

Questions?
Thank you very much!

Federal Aviation
Administration