direction générale
de l‟Aviation civile

direction des services dela Navigation aérienne

direction de laTechnique et del’Innovation

Présenté par

Jean-Marc Loscos

Safety nets performanceassessment: the encounter-model
methodology as a cornerstone toprovide quantified results for
ACAS and STCAEIWAC2010 – Tokyo 10-12 November 2010

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

Contents

direction de la Technique etde l’Innovation• Background on STCA and ACAS
• ACAS performance and standardisation• Encounter models• Overview of the simulation framework• (STCA models, ATC surveillance models, pilot and controller models)• Results on key performance areas of STCA
• Conclusions and future work

-2-

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Background & context• STCA & ACAS are two safety nets of different maturity and scope,developed independently from each other• STCA standardization is under progress in Europe(e.g. high-level EUROCONTROL specifications)– Although not mandatory, STCA is deployed in several States with a wide range of
implementation• ACAS performance-oriented Standards and Recommended Practices(SARPs) are defined at ICAO level– ACAS mandatory carriage exists worldwide and there is a single ACAS compliant
equipment (i.e. TCAS II version 7)-3-

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne-4-direction de la Technique etde l’Innovation

Framework for evaluating the safety
benefits of ACAS
• Safety benefits afforded by ACAS expressed in terms of „risk ratio‟
measured using a set of models replicating the environment in which ACAS
is operating : safety encounter model, altimetry error model and pilot
response model.
Altimetry error
model

Radar
data

Pilot response
model

Safetyencounter
model

ACAS
simulations

Contingency
tree

Underlying
risk

Logic system
risk

Full ACAS
system risk

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne-5-direction de la Technique etde l’Innovation

EUROCONTROL I-AM-SAFE framework• EUROCONTROL I-AM-SAFE framework relied on the safety and ATM
encounter models (used in the ACAS field), a simplified STCA model (basedon EUROCONTROL specifications) and a set of STCA performance metrics

ATM encountermodel

ATM & safety
encountersets
STCA
simulations

ACAS
simulations

STCA
performance
metrics

STCA/TCAS
interaction
metrics

Safety encountermodel

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Relevance of the simplified STCA
model
• Implementation of the reference
STCA model proved to be
operationally realistic despite
its simplicity• Areas of improvement:• Development of optional features described in the reference STCA model (to copewith specific situations like slow closing encounters, etc)
• Use of surveillance model that would be representative of actual
surveillance performances (perfect surveillance was assumed)-6-

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Relevance of the ATM & Safety
encounter model• Safety encounter model proved to be more appropriate than the ATM
encounter model for evaluating the efficacy of STCA
– However, due to its focus on risk-bearing situations, the safety encounter model
exaggerated the alert rates• ATM encounter model proved to be more useful for evaluating compatibilityof STCA with day-to-day operations• Areas of improvement:– Development of an ATM incident-based encounter model (derived from real
incidents observed in European radar data) that would encompass the interest of
the safety and ATM encounter models– Development of a controller intervention model in response to STCA apart fromthe encounter model it-self-7-

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne-8-

direction de la Technique etde l’Innovation

Relevance of the STCA performance
metrics• Alert statistics demonstrated to be influenced by:– Encounter characteristics and STCA configuration• Evaluation of different warning time metrics using either:– „Actual trajectories‟ or „Predicted trajectories‟– „Time of Separation Infringement‟,„Time of separation Violation‟ or„Time of Closest Approach‟alerts• Areas of improvement:– More sophisticated metrics
required to take into accountcontroller reaction in response
to STCA

90s

50000
40000

P redicted TSI

30000

A ctual TSI
P redicted TOV

20000

A ctual TOV
P redicted CP P

10000

A ctual CP P

0
<0

<40

<80

<120

<160

seconds

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne-9-direction de la Technique etde l’Innovation

Enhanced framework for STCA
performance evaluation
ATC incident
data

SSR / RDPS
performancedata

Controller response
model

Radar data

ATM incidentbased
encounter
model

Underlying levelof safety(without STCA
nor ACAS)

Pilot response
model

ATC
surveillance
model

Altimetry
error
model

STCA
simulations

ACAS
simulations

Level of safety
achieved with
STCA

Overall level ofsafety with
ACAS

Safety nets contribution

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Encounter modelling (1/3)• Encounter– Traffic situation (operationally realistic) involving two aircraft• Safety encounter model– Close encounters (with almost no horizontal miss distance) with actual or potential
risk of collision– About 1 close encounter every 6,000 flight-hours (or every
2 days of observation by a typical en-route radar)• ATM encounter model– Encounters occurring in routine operations including ATC intervention to preserve
separation– About 4 encounters per flight-hour ≈ about 18 encounters per sector hour

- 10 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 11 -

direction de la Technique etde l’Innovation

Encounter modelling (2/3)• Modelling of observed encounters (statistical distributions of encounter properties
derived from radar data analysis)
aircraft 2:
aircraft 1:
speed
acceleration

speed
acceleration

vertical manoeuvres:timinginitial and final rates
acceleration
convergenceangle

aircraft 1:
altitudevertical rate

VMD

HMD
turns:
timingheading changebank angle

plan view

aircraft 2:
altitudevertical ratevertical profile

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 12 -

direction de la Technique etde l’Innovation

Encounter modelling (3/3)• Five altitude layers with distinct proportions of aircraft performance classes
18%
15%
12%

Most typicalaircraft classover allaltitudes(medium jet)

9%
6%

Most typical aircraftclass below FL50
(light piston)50-FL L 135 5
t
f
21
00
-F
5
1,0 FL50 5-FL L28 153
F41
FL L215 5-FL
8
F
2
FL

3%
0%
Hig

h
J e J et >t1
Tu
10 perfo
5
Je
r
0
bo
,0
rm
,0
t
Tu
Tu
rbo 5,70 prop 0k g- 00k g anc
em
1
rbo
Pi s
0k
p
g-1 >15 , 00,0
pro ropton
il ita
0
0
5,7
5,0
p<
00
0k
ry
<5
0
kg
g
00
5, 7
,70
jet
0k
k
g-1
g
00
0kg
kg
5,0
00
kg

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 13 -

direction de la Technique etde l’Innovation

Pilot response modelling (to ACAS RA)• Derived from operational airborne recordings• Continuum of responses around ICAO standard response– 5s delay, 0.25g acc., 1,500fpm vs21%

Non responding
pilots18%
15%

Most typical pilot
response(5s delay, 0.15g acc.,
1,300fpm vs)

12%
9%
6%
3%
0%
3s

5s

7s

No
0,22g reac tion
-39 0
0,3
0
0,22g g-220 0fp fpm
m
-22 0
0,15
0
0,22g g -22 00fp fpm
m
-13 0
0,15g
0fpm
1
3 00f
0,15g
pm
-73 0
8s 0,0
f
pm
9g -7
3 0fp
m

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Enhanced modelling and analysis,
synthesis and guidelines (1)• Performance and safety Aspects of STCA – full Study (2008-2010)– 3-year EUROCONTROL project led by Egis Avia (France) in partnership with Deep Blue
(Italy), DSNA (France), and QinetiQ (United Kingdom)• A comprehensive framework, that includes a series of models to simulate
operationally realistic scenarios of SNET environment and use, is now in place• These models consist essentially of– Encounter models, but also
– Models of ACAS and STCA
systems,– Controller and pilot
responses to SNET alerts, and– Aircraft behaviour• The set of available models
covers all key factors
influencing SNET performance- 14 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 15 -

direction de la Technique etde l’Innovation

Enhanced modelling and analysis,
synthesis and guidelines (2)• A range of realistic operational scenarios (with and without ground-based
SNETs) for both TMA and en-route airspace has been defined• With different human behaviours as observed during monitoring activity
Scenario for Human
Performance

controller‟s inputof CFL

controller‟s response to STCA

delay reaction
timetype of AI

use of avoiding
phraseology

pilot response
to AIspilot response
to RAsstandard

100%

average (6s)

H or V

sometimes

standard

standard

standard with nominal
phraseology

100%

average (6s)

H or Vneverstandard

standard

optimal100%

prompt (4s)

H or V

alwaysstandard

standard

relaxed95%

slow (10s)

H or Vneverstandard

standard

typical95%

typicalmix of H and V

sometimes

typicaltypical• CFL stands for Cleared Flight Level

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation- 16 -

Enhanced modelling and analysis,
synthesis and guidelines (3)• Several STCA “families” have been identified during monitoring activity with different
parameters and optional features– More or less time-criticalparameters and more orless reduced separationthresholds– Distinction between“basic”, “standard”
or “advanced”
implementation

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 17 -

direction de la Technique etde l’Innovation

Enhanced modelling and analysis,
synthesis and guidelines (4)• A comparative analysis of STCA performances in terms of likelihood, relevanceand efficacy of alerts, as well as level of interaction with ACAS has been
conducted– To evaluate the influence of key factors like STCA configuration and parameters,
encounters characteristics, CNS characteristics, controller and pilot behaviours in
response to alert
initial severity when there is an STCA
– In support to the100%
determination of
possible minimum /80%
recommended performance60%
requirements for STCA,
last_two_mns
including compatibility
40%
before_last_mn
almost_last_mn
with ACAS
beyond_last_mn
very_last_mn

20%
0%
SC1

SC2

SC3

SC4

SC5

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 18 -

direction de la Technique etde l’Innovation

Enhanced modelling and analysis,
synthesis and guidelines (5)• The operational safety assessment of hazards related to collision prevention by
ATCO assisted by STCA, including undesired interaction with TCAS RA, hasbeen completed– Consolidate event-tree
analysis of hazard effects /severity– To derive safety objectives(following apportionmentof ATM safety targets
compliant with ESARR4)– Consolidate fault-tree analysisof possible causes– To derive possible safety
requirements on STCA

Lack of ATCo
instruction to solvea short-term conflict

OH1

There is a
short-term
conflict

Lack of ATCoreaction to solve a
short-term conflict

STC

NO ATC RESOLUTION

Lack of genuine STCA
alert and ATCo does
not detect the
short-term conf lict

Lack of ATCoreaction to genuine
STCA alert

GATE1

GATE15

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne- 19 -

direction de la Technique etde l’Innovation

Overview of the STCA and ACAS
simulation framework
Encounter
model
Assumptions about airspace characteristics
Generatedencounters

Assumptions about ATC surveillance means

Scenarios

Assumptions about STCA and ACAS systems
Assumptions about Human Performance(controller & pilot)

STCA
simulation

STCA performance
metrics & level of safetywith STCA

Encounterswith the effectof STCATCAS
simulation

Encounterswith the effects of
STCA & TCAS

STCA / ACAS
performance metrics &overall level of safety

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Key performance areas of STCA
• Likelihood of STCA alerts (number of alerts per flight hour)• Operational relevance of STCA alerts• Efficacy of alerts (time remaining for ATCO intervention)• Level of STCA and ACAS interaction (relative timing when the 2 SNET are
triggered)- 20 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Results on the likelihood of STCA
alerts• The ANSP strategy when implementing and fine tuning STCA has a direct
impact on the likelihood of STCA alerts• All investigated STCA configurations show comparable alert rates for themost severe encounters• For less severe encounters, STCA configurations designed for collision
avoidance only show an alert rate 100 less than STCA configurations
designed for « separation protection » as well as « collision avoidance »• All STCA configurations issue unnecessary alerts (no loss of separation).
• The quality of the surveillance data used by STCA also has a small effect on
the STCA alert rate (factor <1.6)- 21 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Results on the operational efficacy
of STCA alerts• The various STCA configurations provide fairly similar WARNING TIME
performances. Optional features (turning prediction filter, use of CFL or
SFL) improve the separation margins in the most time-critical alerts.

• Safety benefits ?
• Ratio of (separation infringements with the effect of STCA) versus(Separation infringements without the effect of STCA.)

The less conservative STCA families appear to be less effective than theother families to maintain or restore separation. However, all but one STCA
family reduce the number of separation infringements for severe encountersby a factor of at least FIVE (Ratio <20%)- 22 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation

Project close-out and further work
– PASS Final study report (November 2010)– PASS Dissemination workshop on 23th of November 2010 (Brussels)
A step further towards a consistent overall concept for ground-based and airborne
safety nets in coordination with appropriate bodies
SESAR Operational Project 4.8.1 (Evolution of Ground-Based Safety Nets) will
use the report and the methodology to express Operational Requirements, Safety
and Performance requirements in support of the development of an industrial
prototype by the SESAR Technical Project 10.4.3 (Safety Nets adaptation to newmodes of operation) in 2010-2011.
 It is anticipated that the operational validation of the STCA prototype would be
conducted in the 4th QTR 2011.

- 23 -

November 2010

direction générale
de l‟Aviation civile

direction des services de laNavigation aérienne

direction de la Technique etde l’Innovation- 24 -